From 968066ff23b084e0a00e968a2f2c37e6ece91801 Mon Sep 17 00:00:00 2001 From: Frank Date: Mon, 17 Aug 2026 03:30:00 -0700 Subject: [PATCH] docs: boot policy (only PP units enabled), deploy.sh dankserver hazard, stale-claim cleanup - deployment.md: Boot Policy (2026-08-17) section (disabled units table, re-enable recipes, stale frank-restart-small.sh helper, broken deploy.sh dankserver target); Update Dankserver -> frank-pp-worker; condensed retired 35B history block and 2026-08-13 incident - AGENTS.md: boot-policy note, target map fix (Q6_K, split 0.68), deploy.sh dankserver warning, obsolete quirk entries condensed - README.md: rewrite active services/models tables (PP head + worker) - index.md: topology fix (qwythos:9b, Q6_K split 0.68, NPU disabled) - qwen3.8-spec-matrix.md: superseded configs merged into compact history; stale inline env block removed; bundled-head MTP note - FastFlowLM, PP Deployment: boot-policy + enablement notes - System Specs x2, DFlash doc, 35B FP8 doc, Ling docs: stale active claims fixed, retired banners, never-shipped plan stubbed --- AGENTS.md | 26 +-- README.md | 42 ++--- docs/DFlash DSpark Speculative Decoding.md | 14 +- docs/Dankclaw System Specs.md | 2 +- docs/Dankserver System Specs.md | 4 +- docs/FastFlowLM NPU Inference.md | 10 ++ .../Qwen3.8 PP Deployment.md | 15 +- ...nning Qwen3.6 35B A3B FP8 on Dankserver.md | 5 +- docs/deployment.md | 79 ++++++--- docs/index.md | 14 +- docs/ling-3.0-flash-trial.md | 19 +-- docs/ling-3.0-flash-vllm-plan.md | 46 +---- docs/qwen3.8-spec-matrix.md | 160 +++++------------- 13 files changed, 190 insertions(+), 246 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 0867f22..25fef98 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -15,11 +15,11 @@ The Dankclaw NPU endpoint (FastFlowLM, `frank-default-small`, port 52625) serves Qwythos-9B-Claude-Mythos-5-1M (abliterated) as registry tag `qwythos:9b` at 262144 ctx (swapped in 2026-08-17; previously `qwen3.6-moe:35b-a3b`; the old llama.cpp 35B service on Dankserver is -stopped). See [docs/deployment.md](docs/deployment.md) for the deployment -process, [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md) for the -spec history and current production state, and -[docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md](docs/Pipeline%20Parallelism/Qwen3.8%20PP%20Deployment.md) -for the PP runbook. +stopped). **Boot policy (2026-08-17):** only `frank-default-large` and +`frank-pp-worker` are enabled for boot. The NPU unit `frank-default-small` +is disabled; the running service keeps serving until stopped or rebooted. +See [docs/deployment.md](docs/deployment.md) for the deployment +process, the boot policy, and the re-enable recipes. **Sibling project:** [../serverAppConfigs](../serverAppConfigs) manages the Podman app stack on the same hosts (OpenWebUI, Gitea, observability). Its @@ -83,10 +83,14 @@ Target map: | Target | Host | Runtime | Service | Port | |--------|------|---------|---------|------| -| Dankclaw | dank@dankclaw | llama.cpp PP head (Qwen3.8-27B UD-Q8_K_XL, split 0.40/0.60 to the Dankserver worker via RPC, MTP+ngram-mod, f16 KV) | frank-default-large | 7770 | -| Dankclaw | dank@dankclaw | FastFlowLM NPU (XDNA2 50 TOPS, qwythos:9b Q4NX 7.76 GB, 262144 ctx; see [[FastFlowLM NPU Inference]]) | frank-default-small | 52625 | +| Dankclaw | dank@dankclaw | llama.cpp PP head (Qwen3.8-27B-Uncensored Q6_K, split 0.68/0.32 to the Dankserver worker via RPC, standalone MTP draft + ngram-mod, q8_0/f16 KV) | frank-default-large | 7770 | +| Dankclaw | dank@dankclaw | FastFlowLM NPU (XDNA2 50 TOPS, qwythos:9b Q4NX 7.76 GB, 262144 ctx; see [[FastFlowLM NPU Inference]]) — disabled for boot since 2026-08-17 | frank-default-small | 52625 | | Dankserver | dank@dankserver | ggml-rpc worker (RTX 3090, receives the head's tensor share) | frank-pp-worker | 50052 | +> Warning: `deploy.sh dankserver` is currently broken. It targets the +> removed `frank-default-small` unit. Restart the worker with +> `sudo systemctl restart frank-pp-worker` until the script is fixed. + ## Runtime & Troubleshooting ### Env var naming rule — HIP runtime collision @@ -117,7 +121,7 @@ graph TD 2. Run the serve script directly. Then run the raw binary with the same flags. A different outcome isolates the env file sourcing. 3. Run gdb for the backtrace with this command: ``` - gdb -batch -ex run -ex "bt 25" --args bash scripts/serve-llama-frank-large.sh + gdb -batch -ex run -ex "bt 25" --args bash scripts/serve-pp-head.sh ``` `llama-server` buffers its timed stdout. On SIGSEGV the log tail is lost. Trust gdb, not the log. 4. Kernel spam about `amdgpu: SVM mapping failed` is a symptom, not the root cause. Investigate other causes before assuming memory pressure. @@ -125,13 +129,13 @@ graph TD ### Operational quirks - `sudo` on both boxes needs a tty. The remote sudo steps in `deploy.sh` fail non-interactively. Use `echo '' | sudo -S `. -- Passwordless restarts: each target has a root-owned helper (`/usr/local/sbin/frank-restart-large.sh` on Dankclaw, `frank-restart-small.sh` on Dankserver) and a sudoers NOPASSWD drop-in (`/etc/sudoers.d/frank-*`). `deploy.sh` prefers the helper. The sudoers rules match commands exactly: no extra arguments (for example, `--no-pager` breaks the `systemctl status` rule). See [docs/deployment.md](docs/deployment.md). +- Passwordless restarts: each target has a root-owned helper and a sudoers NOPASSWD drop-in (`/etc/sudoers.d/frank-*`). The Dankclaw helper `/usr/local/sbin/frank-restart-large.sh` is correct for the PP head. The Dankserver helper `frank-restart-small.sh` is **obsolete** — it restarts the removed `frank-default-small` unit. Restart the worker with `sudo systemctl restart frank-pp-worker`. The sudoers rules match commands exactly: no extra arguments (for example, `--no-pager` breaks the `systemctl status` rule). See [docs/deployment.md](docs/deployment.md). - Never run `pkill -f` with a pattern that appears in your own command line. It kills your ssh session. Use `pkill -x `. - Model load is slow: about 3 minutes warm, 8-17 minutes cold (measured 17 minutes on 2026-08-13 after a cache drop). Health returns `503 Loading model` the whole time. That is normal. Do not kill the process. `deploy.sh` waits up to 20 minutes on Dankclaw (see `HEALTH_WAIT`). - Page-cache state confounds crash tests. Cold-cache loads are slow and survive. Warm-cache loads crash fast. Control for it when testing. `deploy.sh` drops caches on Dankclaw before restart (see `DROP_CACHES`); manual restarts must run `sync && echo 3 | sudo tee /proc/sys/vm/drop_caches` first. - A client abort and a server release look different in the journal. A client abort logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in [docs/deployment.md](docs/deployment.md). -- Dankserver crash signature under parallel load: `CUDA error: an unsupported value or parameter was passed to the function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541`, then `status=6/ABRT`, then `503 Loading model` for ~5 minutes. Happened twice on 2026-08-13 under a 5-researcher subagent storm (llama.cpp b10121, driver 595.84). Root cause not yet determined; do not confuse it with speculative decoding. -- DFlash premature EOS under parallel load: on Dankclaw, `draft-dflash` ends a generating slot ~1s after a new task lands with high prompt-cache reuse (LCP ~0.99), with `finish_reason "stop"` and no `cancel task` log. It is a DFlash shared-draft-context bug. DFlash is retired from Dankclaw. DSpark (`draft-dspark`, its semi-autoregressive successor) is also unsafe here: two concurrent 16k prose prompts crashed the ROCm backend with `ROCm error: an illegal memory access` (status 6/ABRT). Production uses `--spec-default` (ngram-mod, draftless) at PARALLEL=2 on the Vulkan0 backend. See [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md). +- Dankserver crash signature (historical): the retired 35B service aborted under parallel load with `CUDA error: an unsupported value or parameter was passed to the function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541`, then `status=6/ABRT`, then `503 Loading model` for ~5 minutes (llama.cpp b10121, driver 595.84). Root cause was never determined; the service is retired. Do not confuse that crash with speculative decoding. See [docs/dankserver-vram-growth.md](docs/dankserver-vram-growth.md). +- DFlash premature EOS under parallel load (historical): on Dankclaw, `draft-dflash` ended a generating slot ~1s after a new task landed with high prompt-cache reuse (LCP ~0.99), with `finish_reason "stop"` and no `cancel task` log — a DFlash shared-draft-context bug. DFlash and DSpark are retired from Dankclaw. Production uses the standalone MTP draft over the ngram-mod fallback at PARALLEL=1 on ROCm0. See [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md). ### Dankclaw memory model diff --git a/README.md b/README.md index 32b8ebd..517038e 100644 --- a/README.md +++ b/README.md @@ -1,11 +1,13 @@ # vLLM Scripts -Scripts for running LLM inference on two local machines. +Deployment scripts for the LLM inference fleet: a llama.cpp +pipeline-parallel Qwen3.8-27B split across two machines, plus the +FastFlowLM NPU endpoint on Dankclaw. ## Targets | Host | User | Codename | Architecture | -|------|------|----------|-------------| +|------|------|----------|--------------| | dankserver | dank | Dankserver | NVIDIA RTX 3090 + 62 GB RAM | | dankclaw | dank | Dankclaw | AMD Strix Halo APU (shared memory) | @@ -13,30 +15,32 @@ Scripts for running LLM inference on two local machines. | Service | Host | Runtime | Model | Port | |---------|------|---------|-------|------| -| frank-default-large | dankclaw | vLLM-ROCm (CIRU) | Ling-3.0-flash CIRU INT4 MTP | 7770 | -| frank-default-small | dankserver | llama.cpp (ngram-mod) | Qwen3.6-35B-A3B uncensored Q6_K_P | 7770 | +| frank-default-large | dankclaw | llama.cpp PP head (RPC to Dankserver) | Qwen3.8-27B-Uncensored Q6_K, standalone MTP draft | 7770 | +| frank-pp-worker | dankserver | ggml-rpc worker (RTX 3090) | receives the head's tensor share | 50052 | -> The retired ROCmFPX llama.cpp service on Dankclaw was replaced by the vLLM service above. The retired `vllm-awq-qwen36` (AWQ 4-bit) service on Dankserver was replaced by the llama.cpp service above. Legacy unit files remain in `scripts/` for reference. - -## Models - -| Model | Target | Format | -|-------|--------|--------| -| Ling-3.0-flash CIRU INT4 (MTP) | dankclaw | INT4 W4A16 (Quark/vLLM) | -| Qwen3.6 35B A3B uncensored Q6_K_P (ngram-mod speculative decoding) | dankserver | GGUF Q6_K_P (llama.cpp, MoE offload) | -| Qwen3.6 35B A3B AWQ 4bit | dankserver | AWQ (retired) | -| Qwen3.6 35B A3B FP8 | dankserver | FP8 (documented, not active) | -| Qwen3.6 27B FP8 | dankclaw | FP8 (documented, not active) | +Boot policy (2026-08-17): only these two units are enabled. The +FastFlowLM NPU endpoint (`frank-default-small`, qwythos:9b, port 52625) +is disabled for boot; re-enable it from `scripts/frank-default-small.service` +if needed. Retired units: the Dankserver llama.cpp 35B service, +`llamaserver`, `ollama`, `vllm-awq-qwen36`, `vllm-qwen36`, +`frank-large-ling` (Ling trial), and the vLLM-ROCm CIRU path. Their unit +files remain in `scripts/` for reference. ## Structure -- `docs/` — Project knowledge base with research notes on vLLM configuration, hardware specifics, speculative decoding, and deployment notes -- `scripts/` — Deployment and launch scripts (deploy.sh, serve-llama-frank-small.sh, serve-ling3-vllm.sh, systemd unit files) +- `docs/` — knowledge base: deployment runbook, PP runbook, spec matrix, + NPU notes, research archive +- `scripts/` — serve scripts, env files, systemd units, deploy and test + tools ## Deployment -See [docs/deployment.md](docs/deployment.md) for the deploy/update process. Run `./scripts/deploy.sh ` after pushing committed changes. +See [docs/deployment.md](docs/deployment.md). Run +`./scripts/deploy.sh dankclaw` for the PP head. The `dankserver` target +is broken (it targets the removed `frank-default-small` unit); restart +the worker with `sudo systemctl restart frank-pp-worker` until the +script is fixed. ## Knowledge Base -See [docs/index.md](docs/index.md) for the project knowledge base index. \ No newline at end of file +See [docs/index.md](docs/index.md) for the project knowledge base index. diff --git a/docs/DFlash DSpark Speculative Decoding.md b/docs/DFlash DSpark Speculative Decoding.md index d19ce30..8a1f41b 100644 --- a/docs/DFlash DSpark Speculative Decoding.md +++ b/docs/DFlash DSpark Speculative Decoding.md @@ -160,9 +160,7 @@ The DFlash draft is not a standalone model. It must pair with its target model. ## Deployment on Dankserver (NVIDIA RTX 3090) -## Deployment on Dankserver (NVIDIA RTX 3090) - -The active Dankserver deployment runs **llama.cpp** (`frank-default-small`), not vLLM. It serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` on port 7770 using **ngram-mod** speculative decoding (n_min=48 / n_max=64 / n_match=24) with 3 slots at 200000 context each and MoE experts on CPU. See [[deployment]] and `scripts/serve-llama-frank-small.sh`. +The Dankserver llama.cpp deployment (`frank-default-small`, Qwen3.6-35B-A3B) is retired (2026-08-16) and its unit disabled (2026-08-17). The RTX 3090 now runs the PP worker for the Dankclaw head (see [[Qwen3.8 PP Deployment]]). The vLLM material below documents the archived AWQ/DFlash path on the same hardware. > **DFlash retired from active service.** `DFlash` (draft-dflash) was tested initially but did not deliver the desired speed gains under the `frank-default-small` workload (see ngram-mod benchmarks below). The `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` draft GGUF remains in `/home/dank/models/` as reference material only. @@ -197,15 +195,7 @@ docker run --rm --name vllm-qwen36-dflash \ ## Deployment on Dankclaw (AMD Strix Halo gfx1151) -Dankclaw now runs llama.cpp as its runtime for `frank-default-large` (port 7770), serving the unsloth Qwen3.8-27B `UD-Q8_K_XL` model with **no dedicated draft** on the **Vulkan0** backend. - -Current facts (2026-08-15): - -- Main model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (bundled MTP, unused). -- Spec: llama.cpp **`--spec-default`** preset = ngram-mod (n_match 24 / n_min 48 / n_max 64). No draft model. -- **Backend: Vulkan0** — at 262144 ctx (~200k prompt) Vulkan decodes 5.68 t/s vs ROCm 3.83 t/s (48% faster); ROCm wins prefill 119 vs 106 t/s. See [[qwen3.8-spec-matrix]] for the bench table. -- **PARALLEL=2** — passes the 2x32k parallel stream test. -- KV `q8_0` K / `q8_0` V; context 2 x 262144; batch 8192 / ubatch 256. +`frank-default-large` on Dankclaw runs the llama.cpp pipeline-parallel head (Qwen3.8-27B-Uncensored Q6_K) on port 7770, RPC-split with the Dankserver worker. Spec is the standalone MTP draft over the ngram-mod fallback, 1 slot at 262144 ctx, q8_0/f16 KV, head GPU side ROCm0. See [[qwen3.8-spec-matrix]] and [[Qwen3.8 PP Deployment]] for the current state and the 2026-08-15 Vulkan0/ROCm0 high-context bench. History: before `--spec-default`, the box ran DSpark (`draft-dspark`, retired 2026-08-15 after a ROCm illegal-memory-access crash under two concurrent 16k prompts) and before that bundled MTP (`draft-mtp`, retired same day). DFlash was retired on 2026-08-14 (premature-EOS under parallel load). The old DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`), the MTP-only drafter (`Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`), and the DSpark draft (`Qwen3.8-27B-DSpark-Q8_0.gguf`) remain in `/home/dank/models/` as reference only. diff --git a/docs/Dankclaw System Specs.md b/docs/Dankclaw System Specs.md index a913cea..90f4799 100644 --- a/docs/Dankclaw System Specs.md +++ b/docs/Dankclaw System Specs.md @@ -82,7 +82,7 @@ The NPU shares the 30 GiB unified pool. FLM allocates roughly half of system RAM - ROCm reports 96 GiB of GPU-visible memory (overcommitted). The current vLLM service runs at `--gpu-memory-utilization 0.95`, which fits the CIRU INT4 weights and 256k KV cache (about 90 GiB used of the 96 GiB frame buffer). - Without the ROCm VRAM patch, ROCm clamps APU memory to 50% (see `scripts/patches/rocm_patched.py` on the remote). The patch restores the real GTT ceiling minus an 8 GB safety margin. - Only 8 GiB swap. Modern vLLM uses `--kv-offloading-size` instead of the removed `--swap-space` flag. -- The active service on Dankclaw is `frank-default-large.service`: vLLM-ROCm serving the Ling-3.0-flash CIRU INT4 (W4A16) model on port 7770 with native MTP speculative decoding. It replaced the ROCmFPX llama.cpp service. +- The active service on Dankclaw is `frank-default-large.service`: the llama.cpp pipeline-parallel head (Qwen3.8-27B-Uncensored Q6_K) on port 7770, RPC-split with the Dankserver worker. The vLLM-ROCm CIRU Ling path was planned but never shipped. See [[deployment]]. - ROCm 7.2.4 from AMD repos. The CIRU venv bundles a self-contained ROCm 7.15 toolchain; vLLM ROCm support requires ROCm 6.3+. - Python 3.14.4 is very new. vLLM ROCm wheels target Python 3.12. The CIRU venv uses Python 3.12. - Radeon 8060S falls under Ryzen AI MAX series (gfx1151). vLLM lists this as supported. diff --git a/docs/Dankserver System Specs.md b/docs/Dankserver System Specs.md index f5b7b59..af42a51 100644 --- a/docs/Dankserver System Specs.md +++ b/docs/Dankserver System Specs.md @@ -51,11 +51,11 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 24.0 ### Active Services -The active `frank-default-small` service (llama.cpp, `b10108+`) serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model on port 7770 with **ngram-mod** speculative decoding. It runs **4 parallel slots** at 175104 context each (pool 700416, refitted 2026-08-14 for the dual-f16 KV cache). Of the 40 layers' MoE expert weights, **36 sit on CPU** (`--n-cpu-moe 36`) while the remaining **4 layers' expert weights** are resident on GPU along with all non-expert tensors (`--gpu-layers all`). KV stays on GPU with K=`f16`, V=`f16` (13680 MiB pool at 700416 cells). Batch 8192 / ubatch 3072, `--ctx-checkpoints 4`, `--no-mmap`. At idle ~22.1 GB VRAM used, leaving ~2.0 GiB headroom. Idle RAM ~32 GB of 62 GB. +The active service on Dankserver is `frank-pp-worker.service`: the ggml-rpc worker (RTX 3090) that receives the PP head's tensor share (45 of 65 blocks, ~23.8 GiB of 24 GiB at steady state). The old llama.cpp Qwen3.6-35B-A3B service (`frank-default-small`) is retired (2026-08-16) and disabled (2026-08-17). See [[Qwen3.8 PP Deployment]]. ## Hardware Relevance -- 24 GB VRAM is close to saturated with this config. At idle ~22.1 GB used, leaving ~2.0 GiB headroom at `--n-cpu-moe 36` (4 layers' experts on GPU) with `ubatch 3072` and a dual-f16 KV pool of 13680 MiB at 700416 cells (2026-08-14 refit; the same pool at 800768 cells OOM'd the load warmup). A llama-bench sweep ([[frank-default-small-benchmark]]) guided this state: prefill is the priority, so `ubatch 3072` stays (top prefill tier), and `ncmoe 36` was raised from 35 to fix a cuBLAS OOM-class crash during big prefills by freeing ~0.75 GiB peak VRAM at only -2% prefill. If it still crashes under load, add `GGML_CUDA_FORCE_MMQ=1`. If decode speed matters more, use `ncmoe 32 - ubatch 1024` (gen ~48.7 t/s). By editing `scripts/frank-default-small.env`. +- 24 GB VRAM is close to saturated with this config. At idle ~22.1 GB used, leaving ~2.0 GiB headroom at `--n-cpu-moe 36` (4 layers' experts on GPU) with `ubatch 3072` and a dual-f16 KV pool of 13680 MiB at 700416 cells (2026-08-14 refit; the same pool at 800768 cells OOM'd the load warmup). A llama-bench sweep ([[frank-default-small-benchmark]]) guided this state: prefill is the priority, so `ubatch 3072` stays (top prefill tier), and `ncmoe 36` was raised from 35 to fix a cuBLAS OOM-class crash during big prefills by freeing ~0.75 GiB peak VRAM at only -2% prefill. If it still crashes under load, add `GGML_CUDA_FORCE_MMQ=1`. If decode speed matters more, use `ncmoe 32 - ubatch 1024` (gen ~48.7 t/s). The tuning guidance applied to the retired `frank-default-small` service; the PP worker's allocations are fixed at load. - 62 GB system RAM holds the ~29 GB of MoE expert weights with room to spare (~57 GB available after load). - 127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled. - Root disk has approximately 140 GB free. Store the GGUF models under `/home/dank/models` (about 32 GB total). diff --git a/docs/FastFlowLM NPU Inference.md b/docs/FastFlowLM NPU Inference.md index e59b853..e4535cd 100644 --- a/docs/FastFlowLM NPU Inference.md +++ b/docs/FastFlowLM NPU Inference.md @@ -200,6 +200,16 @@ Notes: `frank-default-small` now runs the NPU endpoint on Dankclaw (was the Dankserver llama.cpp service). Unit: `scripts/frank-default-small.service` (symlinked into `/etc/systemd/system`), wrapper `scripts/serve-flm-small.sh`, tunables `scripts/frank-default-small.env`. `LimitMEMLOCK=infinity` is set in the unit. Default model: `gemma4-it:e4b`. Swap `FLM_MODEL` in the env to switch. +### Boot policy (2026-08-17) + +The `frank-default-small` unit is disabled for boot. Only the PP pair (`frank-default-large` + `frank-pp-worker`) starts at boot. The unit file was removed from `/etc/systemd/system`; the running FLM service keeps serving until stopped or rebooted. To restore the endpoint, run these commands on Dankclaw: + +``` +sudo ln -sf /home/dank/vllmScripts/scripts/frank-default-small.service /etc/systemd/system/frank-default-small.service +sudo systemctl daemon-reload +sudo systemctl enable frank-default-small +``` + Live config (2026-08-16): `FLM_PMODE=turbo`, `FLM_ASR=1` (loads Whisper-V3-Turbo-NPU2; exposes `/v1/audio/transcriptions`), `FLM_CTX_LEN=131072` (verified max — Gemma4 natively supports 128K and the NPU budget fits it; KV occupancy measured at 0.11% for a 123-token exchange, decode 12.0 t/s, 13 GiB headroom). The PP split was renamed `frank-pp-head` → `frank-default-large` (2026-08-16, commit `48142f5`). diff --git a/docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md b/docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md index ff0e9b4..8fd60ca 100644 --- a/docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md +++ b/docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md @@ -1,6 +1,6 @@ # Qwen3.8-27B Pipeline-Parallel Deployment (Dankclaw + Dankserver) -Split Qwen3.8-27B UD-Q8_K_XL across the Strix Halo iGPU (head) and the RTX +Split Qwen3.8-27B across the Strix Halo iGPU (head) and the RTX 3090 (worker) using llama.cpp RPC. This page is the design record and the morning-of runbook. Research background: [Qwen3.8 Strix optimizations](../../qwen3.8-strix-optimizations.md). @@ -31,6 +31,19 @@ state, verified against the live journal and output smoke tests - Unit `frank-default-large.service`, launched through `serve-pp-head.sh` + `frank-default-large.env`. +## Service enablement (2026-08-17) + +The PP pair is the only inference service enabled for boot. Both units +(`frank-default-large` on Dankclaw, `frank-pp-worker` on Dankserver) are +enabled. The NPU unit (`frank-default-small` on Dankclaw) and all retired +units are disabled. Restart order: worker first, then head. The Dankserver +helper `frank-restart-small.sh` is obsolete (it restarts the removed +`frank-default-small` unit); restart the worker with +`sudo systemctl restart frank-pp-worker`. The Dankclaw helper +`frank-restart-large.sh` (sync + cache drop + restart) is correct for the +head. `deploy.sh dankserver` is broken until retargeted. See +[[deployment]] for the full boot-policy table. + ## FA-all-quants rebuild (2026-08-17) Flash Attention with mismatched K/V cache types requires the diff --git a/docs/Running Qwen3.6 35B A3B FP8 on Dankserver.md b/docs/Running Qwen3.6 35B A3B FP8 on Dankserver.md index 32cbc20..d0ce93d 100644 --- a/docs/Running Qwen3.6 35B A3B FP8 on Dankserver.md +++ b/docs/Running Qwen3.6 35B A3B FP8 on Dankserver.md @@ -1,5 +1,7 @@ # Running Qwen3.6 35B A3B FP8 on Dankserver +> **RETIRED.** This page documents the FP8 vLLM path for Qwen3.6-35B-A3B on Dankserver. The llama.cpp 35B service is retired (2026-08-16), its unit file was removed (2026-08-17), and the FP8 vLLM unit (`vllm-qwen36`) never shipped and is disabled. The content is kept as research history. The RTX 3090 now runs the PP worker: see [[Qwen3.8 PP Deployment]]. + ## Model Information Qwen3.6-35B-A3B is a Mixture-of-Experts vision-language model from the Qwen3.6 series. It uses 35B total parameters with only 3B activated per token. @@ -112,10 +114,9 @@ export HF_HOME=/mnt/entropy/huggingface ### Active Service Conflict -The active `frank-default-small` service (llama.cpp, port 7770) uses about 22 GB of the 24 GB VRAM. The retired `vllm-awq-qwen36` (Podman, ~22 GB) and `vllm-qwen36` services are not active. Stop any competing service before starting another: +The retired `frank-default-small` service (llama.cpp) used about 22 GB of the 24 GB VRAM. The current PP worker holds ~23.8 GiB. The `vllm-awq-qwen36` (Podman) and `vllm-qwen36` units are disabled. Before starting any research service, verify that the GPU is free: ``` -podman stop vllm-qwen36-dflash nvidia-smi # verify GPU is free ``` diff --git a/docs/deployment.md b/docs/deployment.md index 020b561..57d0799 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -8,11 +8,43 @@ Two servers run LLM inference services. Each server pulls its scripts from this | Target | Host | Runtime | Service | Port | |--------|------|---------|---------|------| -| Dankclaw | dank@dankclaw | llama.cpp PP head (Qwen3.8-27B UD-Q8_K_XL) | frank-default-large | 7770 | +| Dankclaw | dank@dankclaw | llama.cpp PP head (Qwen3.8-27B-Uncensored Q6_K, split 0.68 to the Dankserver worker) | frank-default-large | 7770 | | Dankclaw | dank@dankclaw | FastFlowLM NPU (qwythos:9b, 262144 ctx) | frank-default-small | 52625 | | Dankserver | dank@dankserver | ggml-rpc worker (RTX 3090) | frank-pp-worker | 50052 | -The `frank-default-small` (NPU) unit is not in `deploy.sh`. Restart it by hand with `sudo systemctl restart frank-default-small`. +The `frank-default-small` (NPU) unit is not in `deploy.sh` and is disabled for boot (see Boot Policy below). If the unit is still loaded (pre-reboot), restart it by hand with `sudo systemctl restart frank-default-small`. After a reboot, re-enable the unit first (see Boot Policy). + +## Boot Policy (2026-08-17) + +Only the pipeline-parallel pair starts at boot: `frank-default-large` (Dankclaw head) and `frank-pp-worker` (Dankserver worker). Every other inference unit is disabled. This was set on 2026-08-17. + +Disabled units: + +| Machine | Unit | Fate | +|---------|------|------| +| Dankclaw | `frank-default-small` (FastFlowLM NPU, port 52625) | Disabled; unit file removed from `/etc/systemd/system`. The service keeps running until stopped or rebooted. | +| Dankclaw | `frank-large-ling` (Ling trial) | Disabled. | +| Dankclaw | `vllm-thinkingcap` | Disabled (already). | +| Dankserver | `frank-default-small` (old llama.cpp 35B) | Disabled; unit file removed. | +| Dankserver | `llamaserver`, `ollama`, `vllm-awq-qwen36`, `vllm-qwen36` | Disabled. | + +`systemctl disable` removes the symlinked unit file from `/etc/systemd/system` when the file is a symlink. The repo copies stay intact. To restore a disabled service, re-symlink the unit and enable it: + +``` +sudo ln -sf /home/dank//scripts/.service /etc/systemd/system/.service +sudo systemctl daemon-reload +sudo systemctl enable +``` + +The NPU re-enable recipe uses the Dankclaw repo path `/home/dank/vllmScripts/scripts/frank-default-small.service`. + +The Dankserver helper `frank-restart-small.sh` targets the removed `frank-default-small` unit. Do not use it. Restart the PP worker with an explicit command: + +``` +sudo systemctl restart frank-pp-worker +``` + +`scripts/deploy.sh dankserver` also targets the removed unit. Do not run it until the script is retargeted to `frank-pp-worker`. The `dankclaw` target works: it restarts the PP head through `frank-restart-large.sh`. ## Workflow @@ -77,7 +109,7 @@ Each target has a root-owned helper script that performs the full restart. The u | Target | Helper | Sudoers drop-in | |--------|--------|-----------------| | Dankclaw | `/usr/local/sbin/frank-restart-large.sh` | `/etc/sudoers.d/frank-large` | -| Dankserver | `/usr/local/sbin/frank-restart-small.sh` | `/etc/sudoers.d/frank-small` | +| Dankserver | `/usr/local/sbin/frank-restart-small.sh` (obsolete, see below) | `/etc/sudoers.d/frank-small` | The Dankclaw helper does these steps: @@ -87,7 +119,7 @@ The Dankclaw helper does these steps: 4. Reloads systemd. 5. Enables and restarts the service. -The Dankserver helper does the same steps without the cache drop. +The Dankserver helper `frank-restart-small.sh` is obsolete: it restarts the removed `frank-default-small` unit. Do not use it. The sudoers drop-in does not cover `frank-pp-worker`, so the worker restart needs the password (or a new helper). Restart the worker with `sudo systemctl restart frank-pp-worker`. The sudoers drop-in contains exactly two rules: @@ -125,17 +157,24 @@ curl -sf http://127.0.0.1:7770/health ### Update Dankserver -Run these commands on Dankserver: +Run these commands on Dankserver to update and restart the PP worker: ``` cd /home/dank/vLLM git pull origin main -sudo ln -sf /home/dank/vLLM/scripts/frank-default-small.service /etc/systemd/system/frank-default-small.service +sudo ln -sf /home/dank/vLLM/scripts/frank-pp-worker.service /etc/systemd/system/frank-pp-worker.service sudo systemctl daemon-reload -sudo systemctl restart frank-default-small -curl -sf http://127.0.0.1:7770/health +sudo systemctl restart frank-pp-worker ``` +The worker has no HTTP health endpoint. Verify the listener instead: + +``` +ss -tln | grep :50052 +``` + +The old passwordless helper `frank-restart-small.sh` targets the removed `frank-default-small` unit. Do not use it. See Boot Policy. + ## Service Units ### Dankclaw — frank-default-large @@ -189,15 +228,9 @@ layer offload (`--gpu-layers all`, `--fit off`), batch 8192 / ubatch > table. Also retired earlier: the vLLM-ROCm CIRU INT4 Ling-3.0-flash > runtime, and the llama-serve/vLLM units on Dankclaw. -### Dankserver — frank-default-small (STOPPED 2026-08-16) +### Dankserver — old llama.cpp 35B service (retired 2026-08-16) -> Note (2026-08-16): the `frank-default-small` alias now belongs to the FastFlowLM NPU endpoint on Dankclaw (port 52625, see [[FastFlowLM NPU Inference]]). The llama.cpp Qwen3.6-35B-A3B service on Dankserver described below is stopped (verified `inactive` 2026-08-16). The text below is kept as history. - -The active service on Dankserver runs llama.cpp (upstream, `b10108+`) and serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model under the alias `frank-default-small` on port 7770. It uses **ngram-mod** speculative decoding (`--spec-ngram-mod-n-min 48 / n_max 64 / n_match 24`), **4 parallel slots** at 175104 context each (pool 700416), and mmproj vision. **36 of 40 MoE expert layers** stay on CPU (`--n-cpu-moe 36`); 4 expert layers' weights are back on GPU alongside all non-expert tensors (`--gpu-layers all`). KV stays on GPU (K=`f16`, V=`f16`), refitted from 800000 on 2026-08-14 when the pool went dual-f16 (the f16/f16 pool costs 13680 MiB at 700416 cells and did not fit at 800768 cells). Batch 8192 / ubatch 3072, `--ctx-checkpoints 4`, `--no-mmap`. The unit file is `scripts/frank-default-small.service`, symlinked to `/etc/systemd/system/`. It launches through `scripts/serve-llama-frank-small.sh`, which reads tunables from `scripts/frank-default-small.env`. - -> Note: the Dankserver process currently observed runs `--spec-type ngram-mod` with `--parallel 4`, batch 8192 / ubatch 2048, and `--n-cpu-moe 34` (this is being bench-tuned by the operator; reconcile against `scripts/frank-default-small.env`). - -> Note: this service replaced the retired `vllm-awq-qwen36` (AWQ 4-bit, port 34401) and the earlier `vllm-qwen36` (FP8) / `llama-server` (ROCmFPX) units. The repo tracks `main` on Dankserver at `/home/dank/vLLM`. +Dankserver previously ran `frank-default-small`: llama.cpp serving the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` (ngram-mod spec, 4 slots x 175104 ctx, 36 of 40 MoE expert layers on CPU, mmproj vision). The service was verified `inactive` on 2026-08-16, and its unit file was removed on 2026-08-17 under the boot policy. The name `frank-default-small` now belongs to the FastFlowLM NPU endpoint on Dankclaw (see [[FastFlowLM NPU Inference]]). The RTX 3090 now runs the PP worker. See git history for the full retired configuration. ## Validation @@ -230,17 +263,11 @@ If a service segfaults at load with no obvious cause, compare the exported env a ## Incident: Jobs "Released Prematurely" Under Parallel Load (2026-08-13) -The user reported that on `frank-default-large` (Dankclaw), a job that is streaming tokens gets released early when another job starts streaming. The suspicion was the speculative decoding configuration ("dspark"). Investigation found no dspark in either deployment: Dankclaw runs `draft-dflash`, Dankserver runs `ngram-mod`. Three findings emerged: +A user reported that a streaming job on `frank-default-large` was released early when another job started. The suspicion was speculative decoding. The investigation found three facts: -1. **The server does not preempt.** `llama-server` defers a new task into a queue when all slots are busy. It never releases a running generation to make room. Verified empirically: two concurrent streaming jobs completed normally with dissimilar prompts, similar prompts, and near-identical prompts (high LCP reuse). `parallel=3` on Dankclaw serves up to three concurrent jobs and queues the rest. -2. **The releases were client-side aborts.** The pi coding agent aborts a stream when the user interrupts it (for example, Escape, or starting a new message in a way that cancels the previous one). The client records `stopReason: "aborted"` with `errorMessage: "Operation aborted"`, and the server logs `srv stop: cancel task, id_task = N`. The server journal shows a `slot release` line with no `cancel task` line when the generation ends normally (EOS or limit). -3. **Dankserver crashed under the parallel researcher load.** Five parallel research subagents hammered `frank-default-small` with back-to-back requests (a new task every few seconds for ~15 minutes). At 19:45:44 and again at 19:55:27 the service aborted with `CUDA error: an unsupported value or parameter was passed to the function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541`. systemd restarted it, and in-flight streams died with `Connection error.` or `terminated`, then `503 Loading model` during the ~5 minute reload. This crash, not speculative decoding, killed the parallel jobs. The signature is a CUBLAS invalid-value error in a GEMM under heavy concurrent load; root cause on Dankserver (llama.cpp b10121, driver 595.84, MoE experts CPU-offloaded) was not yet determined. - -Diagnosis recipe: to tell a client abort from a server release, read the journal around the release time. A client abort shows `W srv stop: cancel task, id_task = N`. A natural stop shows only `slot release` with `truncated = 0` and the final timings line. - -### VRAM and RAM growth under load - -This crash class recurs on Dankserver. The 2026-08-14 investigation ruled out ctx-checkpoints and ngram-mod as VRAM growth sources and confirmed RAM growth. See [[dankserver-vram-growth]] for the full findings, the monitoring setup, and the mitigation options. +1. **The server does not preempt.** `llama-server` defers a new task into a queue when all slots are busy. It never releases a running generation to make room. Verified empirically with concurrent streaming jobs. +2. **The releases were client-side aborts.** The pi coding agent aborts a stream when the user interrupts it. The client records `stopReason: "aborted"`; the server logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. +3. **Dankserver crashed under the parallel researcher load.** Five parallel research subagents hammered `frank-default-small` with back-to-back requests. The service aborted twice with `CUDA error: an unsupported value or parameter was passed to the function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541`, then `status=6/ABRT`, then `503 Loading model` for ~5 minutes. That crash, not speculative decoding, killed the parallel jobs. The affected service is retired; see [[dankserver-vram-growth]] for the VRAM/RAM growth findings behind the crash class. ## See Also diff --git a/docs/index.md b/docs/index.md index a172883..8b19407 100644 --- a/docs/index.md +++ b/docs/index.md @@ -19,21 +19,23 @@ frank-pp-worker, port 50052] D[gfx1151 APU 30GiB phys / 96GiB GPU-visible] E[llama.cpp PP head frank-default-large, port 7770] - F[Qwen3.8-27B UD-Q8_K_XL -split 0.40/0.60 to worker, MTP+ngram-mod] + F[Qwen3.8-27B-Uncensored Q6_K +split 0.68/0.32 to worker, standalone MTP draft + ngram-mod] D --> E --> F G[FastFlowLM NPU XDNA2 50 TOPS -frank-default-small, port 52625] - H[qwen3.6-moe:35b-a3b -12.5-13.1 t/s, 262k ctx, vision] +frank-default-small, port 52625 - DISABLED for boot 2026-08-17] + H[qwythos:9b +Q4NX 7.76 GB, 262k ctx] D --> G --> H end I[LLM Clients / API] -->|HTTP :7770| E I -->|HTTP :52625| G - E -->|RPC layers 0.40| B + E -->|RPC layers 0.68| B ``` +Boot policy (2026-08-17): only `frank-default-large` and `frank-pp-worker` are enabled. The NPU endpoint and all retired units are disabled; see [[deployment]] for the table and re-enable recipes. + ## Servers | Server | GPU | VRAM / Memory | OS | diff --git a/docs/ling-3.0-flash-trial.md b/docs/ling-3.0-flash-trial.md index 254c74b..3c8774a 100644 --- a/docs/ling-3.0-flash-trial.md +++ b/docs/ling-3.0-flash-trial.md @@ -1,5 +1,7 @@ # Ling-3.0-flash Q6_K Trial on Dankclaw +> **RETIRED 2026-08-17.** The `frank-large-ling` unit is disabled under the boot policy. `frank-default-large` now runs the Qwen3.8-27B PP head, not the Qwen3.6-27B config described in the (obsolete) switch-back section below. + This page records the Ling-3.0-flash trial that temporarily replaced `frank-default-large` on Dankclaw. It replaces the Qwen3.6-27B service on port 7770 under the same API alias. @@ -83,18 +85,13 @@ B) switch back to Qwen3.6-27B (frank-default-large, MTP n_max 3). Passwordless management is installed: `/usr/local/sbin/frank-restart-ling.sh` and `/etc/sudoers.d/frank-ling`. -## Switch Back to Qwen3.6-27B (MTP) +## Switch Back (obsolete) -The Qwen config was not changed. It remains committed as -`frank-default-large.service` / `serve-llama-frank-large.sh` / -`frank-default-large.env` (draft-mtp, n_max 3, ubatch 8192). - -Run these commands: - -``` -sudo systemctl stop frank-large-ling -sudo systemctl start frank-default-large -``` +The switch-back commands below are obsolete. `frank-default-large` now +runs the Qwen3.8-27B pipeline-parallel head. To restore a Ling trial, +re-enable `frank-large-ling` from the repo unit +(`scripts/frank-large-ling.service`) and serve script +(`scripts/serve-ling-flash.sh`), then start it on a non-conflicting port. ## Caveats diff --git a/docs/ling-3.0-flash-vllm-plan.md b/docs/ling-3.0-flash-vllm-plan.md index 9b22a3e..1baa9d8 100644 --- a/docs/ling-3.0-flash-vllm-plan.md +++ b/docs/ling-3.0-flash-vllm-plan.md @@ -1,40 +1,10 @@ -# Ling-3.0-flash on Dankclaw — vLLM-ROCm Deployment Plan +# Ling-3.0-flash on Dankclaw — vLLM-ROCm Deployment Plan (never shipped) -## Decisions (confirmed) +This page was the plan for a vLLM-ROCm deployment of +`jcbtc/Ling-3.0-Flash-CIRU-int4-Strix-native` (INT4 W4A16, ~62 GB) on +Dankclaw. The deployment never shipped. The actual Ling work used the +llama.cpp Q6_K fork trial instead: [[ling-3.0-flash-trial]]. -| Decision | Choice | -|----------|--------| -| Objective | Quality-first; correctness (tool calling) over raw quant speed | -| Runtime | vLLM-ROCm on host ROCm venv (jcbtc validated gfx1151 path) | -| Model | jcbtc/Ling-3.0-Flash-CIRU-int4-Strix-native (INT4 W4A16, ~62 GB) | -| Accuracy | INT4 primary; MXFP4 (olka-fi) as controlled A/B after stability | -| KV cache | Full precision (no FP8 KV) — quality | -| Speculative decoding | MTP (native, lossless) | -| Context | Native 256k; user cap ~200k; 4-6 concurrent sessions (~8 max_num_seqs) | -| Throughput | Maximum (chunked prefill, large max_num_batched_tokens) | -| systemd unit | frank-default-large.service | -| Served alias | frank-default-large | -| Port | 7770 (8080s overused) | -| Migration | Kill llama.cpp, disable llama-server.service, reclaim memory | -| Delivery | Follow AGENTS.md git workflow; scripts/ + docs/ in repo | - -## Why not alternatives - -- FP8: broken/lossy on Strix Halo; no ROCmFP8 Ling model; 124 GB does not fit. -- HIGHER bpw (Q5/Q6/Q8/BF16/FP8): exceed 96 GB usable wall for a 124B MoE. -- ROCmFP4 (raulvidis): defective tool calling; llama.cpp-only concurrency limits. -- Vulkan: llama.cpp backend only, not vLLM; no quality ceiling unlock. -- FP8 KV cache: rejected for quality (full-precision KV affordable because - Ling is hybrid linear-attention with small KV footprint). - -## Steps - -1. Pre-flight: disk space, ROCm/amdgpu state, memory baseline. -2. Bring jcbtc scripts + frank-default-large.service into repo; commit. -3. Build ROCm vLLM (install-host-deps.sh + build-vllm-gfx1151.sh). -4. Download weights via download-model.sh (~62 GB). -5. Validate on staging: MTP engages, tool calling works, quality + throughput. -6. Cut over: kill llama.cpp, disable llama-server, enable frank-default-large on 7770. -7. Tune max throughput; confirm concurrent-token capacity. -8. MXFP4 A/B (post-stable). -9. Update AGENTS.md, README, and KB. +The trial service (`frank-large-ling`) is retired and disabled under the +boot policy (2026-08-17). `frank-default-large` is the Qwen3.8-27B PP +head, not the Ling path. See git history for the full decision table. diff --git a/docs/qwen3.8-spec-matrix.md b/docs/qwen3.8-spec-matrix.md index e2d1eb3..45ae763 100644 --- a/docs/qwen3.8-spec-matrix.md +++ b/docs/qwen3.8-spec-matrix.md @@ -1,9 +1,8 @@ -# Qwen3.8-27B UD-Q8_K_XL Speculative-Decoding Matrix +# Qwen3.8-27B Speculative-Decoding Matrix -This page records the speculative-decoding benchmark and the production -tuning for the unsloth Qwen3.8-27B `UD-Q8_K_XL` service -(`frank-default-large`), now a pipeline-parallel head with a Dankserver -worker. +This page records the speculative-decoding benchmark history and the +production tuning for the Qwen3.8-27B service (`frank-default-large`), +now a pipeline-parallel head with a Dankserver worker. ## Production state (2026-08-17: Q6_K, q8_0/f16 KV, split 0.68) @@ -42,99 +41,44 @@ Current production, verified against the live process and journal - Batch 8192 / ubatch 256, `--direct-io`, `--fit off`, `--gpu-layers all`. -## Result (superseded 2026-08-17: Q8_K_XL, f16/f16 KV, split 0.53) - -Superseded for topology on 2026-08-16, when the PP split became -production. The ngram-mod preset survives as the fallback layer under -MTP. The Vulkan0 verdict in this section applies to the standalone -configuration only. - -- Model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB, bundled MTP). -- Spec: llama.cpp **`--spec-default`** preset = **ngram-mod** (n_match 24, - n_min 48, n_max 64). No draft model. Chosen 2026-08-15 after DSpark - crashed under parallel load (see below). -- **PARALLEL=2** — passes the 2x32k parallel stream test (both streams - complete 256/256 concurrently, no crash). ngram-mod is draftless and - parallel-safe. -- **Backend: Vulkan0** — at 262144 ctx (~200k prompt) Vulkan decodes - 5.68 t/s vs ROCm 3.83 t/s (48% faster); ROCm only wins prefill (119 vs - 106 t/s). See the high-context bench table below. -- Batch 8192 / ubatch 256, ctx 524288 (262144 per slot). - -## Result (superseded 2026-08-15: DSpark) - -- Model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB, bundled MTP). -- Spec: **DSpark sidecar draft** `magnitudedev/Qwen3.8-27B-DSpark-GGUF` - (`Qwen3.8-27B-DSpark-Q8_0.gguf`, 1.36B, block_size 7), `--spec-type - draft-dspark`, `--spec-draft-n-max 7` (clamps to block_size). -- **PARALLEL=1.** Two concurrent 16k prose prompts crashed the ROCm backend - with `ROCm error: an illegal memory access` during per-sequence block - sampling. See the DSpark parallel incident below. -- Retired the same day in favor of `--spec-default`. -- Batch 8192 / ubatch 256, ctx 524288 (262144 per slot). - -## Result (superseded 2026-08-14: bundled MTP) - -- Model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB, bundled MTP). -- Spec: `draft-mtp`, `--spec-draft-n-max 5`, no `--spec-draft-p-min`. -- Batch 8192 / ubatch 256, parallel 2, ctx 524288 (262144 per slot). -- Realistic-prompt generation: ~13.3 t/s single, ~8 t/s per slot under - 2 concurrent streams. No premature stop. - -## Incident: DSpark ROCm Crash Under Parallel Load (2026-08-15) - -A two-stream parallel acceptance test crashed `frank-default-large`. - -```mermaid -graph TD - A[Two concurrent 16k prose prompts
high prompt-cache reuse] --> B[ROCm illegal memory access
in common_speculative_impl_draft_dflash::draft] - B --> C[status 6/ABRT core dump] - C --> D[systemd auto-restart, 503 during reload] - D --> E[pinned PARALLEL=1] -``` - -Trigger: two distinct-but-overlapping ~16k-token realistic prose prompts -started simultaneously on PARALLEL=2. The ROCm backend aborted at -`ggml-cuda.cu:106` with `ROCm error: an illegal memory access was -encountered`, surfacing in `hipStreamSynchronize` inside -`common_speculative_impl_draft_dflash::draft` (the class used by -`draft-dspark`). `systemd` caught it (status 6/ABRT) and restarted the -unit; in-flight streams saw `Remote end closed connection without response` -until the ~reload finished. - -Mechanism: DSpark reuses the DFlash block-diffusion draft machinery. The -DFlash draft model uses one shared draft context (`ctx_dft`) across all -parallel slots; under concurrent block drafting on ROCm it performs an -illegal memory access (HIP stream sync). This is the same multi-slot -class of bug that retired `draft-dflash` for premature EOS; here it is a -hard ROCm crash instead of a bad sample. - -Fix: pin **PARALLEL=1** (single slot). With one slot, concurrent requests -queue safely and each completes to `max_tokens`. Revisit after an upstream -multi-slot DSpark fix (DSpark is known-good single-slot). Single-slot 16k -prose generation was ~steady (measuring only the correct-size signal, not -this page). - -## Parallel results by stream count (2026-08-15, correct-size tests) - -| Parallel | Two 16k prose prompts | Two 67k prose prompts | -| -------- | -------------------- | --------------------- | -| 2 | crashes (illegal mem access) | passes | -| 1 | queues, both complete | n/a | - -Note: the crash manifests on the smaller (16k) concurrent streams, not the -larger 67k pair. Size is not the trigger; concurrent block drafting on the -shared draft context is. - -## How MTP works on Qwen3.8 +## How MTP works on Qwen3.8 (bundled-head era) The unsloth `UD-*` GGUFs keep the MTP (Multi-Token Prediction) head inside the model file. llama.cpp b10411+ auto-detects it and creates the draft -context from the target model. Do not pass `--spec-draft-model`. +context from the target model. Do not pass `--spec-draft-model` for those +files. This is superseded: the production `Qwen3.8-27B-Uncensored-Q6_K.gguf` +has NO bundled head, and the serve script passes the standalone draft +sidecar with `--spec-draft-model` (see Production state above). The stock llama.cpp build must be b10411 or newer. The 2026-08-14 rebuild on Dankclaw is b10433 (`9b05354ec`) with `GGML_HIP` + `gfx1151` + Vulkan. +## History (superseded configurations, 2026-08-14 to 08-16) + +All superseded configs used `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB, +bundled MTP) before the Q6_K swap. The ngram-mod preset survives as the +fallback layer under MTP. The Vulkan0 verdict from this era applies to the +standalone configuration only; the PP head runs ROCm0. + +- **2026-08-14: bundled MTP** — `draft-mtp`, n_max 5, parallel 2, ctx + 524288 (2 x 262144). ~13.3 t/s single, ~8 t/s per slot under 2 + concurrent streams. No premature stop. +- **2026-08-15: DSpark sidecar** — `magnitudedev/Qwen3.8-27B-DSpark-GGUF` + (Q8_0, 1.36B, block_size 7), n_max 7, PARALLEL=1. Two concurrent 16k + prose prompts crashed the ROCm backend with `ROCm error: an illegal + memory access` in `common_speculative_impl_draft_dflash::draft` (status + 6/ABRT, systemd restart, ~503 during reload). DSpark reuses the DFlash + block-diffusion machinery, which shares one draft context across + parallel slots; concurrent block drafting faults in HIP stream sync. + This is the same multi-slot bug class that retired `draft-dflash` + (premature EOS). The crash shows on 16k concurrent streams, not on a + 67k pair — concurrency is the trigger, not size. Fix: PARALLEL=1 + (requests queue safely). Retired the same day. +- **2026-08-15 to 08-16: `--spec-default` (ngram-mod), Vulkan0** — n_match + 24 / n_min 48 / n_max 64, draftless, PARALLEL=2 (2 x 32k parallel stream + test passes). Chosen for parallel safety after DSpark; kept as the + fallback under MTP in the current config. + ## Benchmark Method The matrix ran 54 configurations: 3 spec types (`draft-mtp,ngram-mod`, @@ -189,37 +133,19 @@ ubatch 256, single slot, on scratch port 7892. Verdict: at very high context, Vulkan0 decodes **48% faster** (5.68 vs 3.83 t/s); ROCm0 prefills 12% faster. Generation is the interactive cost -(users stream tokens), so **Vulkan0** is the production backend. Both -`tg_3s` values were rock-steady across the run (3.83 and 5.67 t/s). +(users stream tokens). This verdict applies to the standalone +configuration; the PP head runs ROCm0 (HIP registers before Vulkan). +Both `tg_3s` values were rock-steady across the run (3.83 and 5.67 t/s). High-context decode collapses on both backends versus ~25 t/s at 16k ctx; that is the long-context attention penalty, not a spec artifact. ## Production Configuration -`scripts/frank-default-large.env` (live 2026-08-16): - -```ini -RPC_SERVERS=192.168.0.235:50052 # Dankserver worker -TENSOR_SPLIT=0.40,0.60 # [RPC0, iGPU] -CTX_SIZE=401400 # pool = 2 slots x 200700 -PARALLEL=2 -SPEC_NONE=0 -SPEC_DEFAULT=1 # ngram-mod fallback (48/64/24) -MTP=1 # draft-mtp, bundled head -SPEC_DRAFT_N_MAX=4 -SPEC_NGRAM_MOD_N_MIN=48 -SPEC_NGRAM_MOD_N_MAX=64 -SPEC_NGRAM_MOD_N_MATCH=24 -CACHE_TYPE_K=f16 -CACHE_TYPE_V=f16 -BATCH_SIZE=8192 -UBATCH_SIZE=256 -``` - -There is no `DEVICE` variable. The head must not pass `--device` (the RPC -device would be silently dropped). Live slot context: `n_ctx_slot = -200704`. The pi `models.json` contextWindow for `frank-default-large` -declares 200000, a safe under-estimate of the live slot. +The live tunables live in `scripts/frank-default-large.env` (head) and +`scripts/frank-pp-worker.env` (worker) in this repo. The Production +state section at the top of this page records the current values: Q6_K +model + standalone MTP draft, split 0.68/0.32, 262144 ctx, PARALLEL=1, +q8_0/f16 KV. ## See Also