dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: boot policy (only PP units enabled), deploy.sh dankserver hazard, stale-claim cleanup

Browse Source 
- deployment.md: Boot Policy (2026-08-17) section (disabled units table,
  re-enable recipes, stale frank-restart-small.sh helper, broken
  deploy.sh dankserver target); Update Dankserver -> frank-pp-worker;
  condensed retired 35B history block and 2026-08-13 incident
- AGENTS.md: boot-policy note, target map fix (Q6_K, split 0.68),
  deploy.sh dankserver warning, obsolete quirk entries condensed
- README.md: rewrite active services/models tables (PP head + worker)
- index.md: topology fix (qwythos:9b, Q6_K split 0.68, NPU disabled)
- qwen3.8-spec-matrix.md: superseded configs merged into compact
  history; stale inline env block removed; bundled-head MTP note
- FastFlowLM, PP Deployment: boot-policy + enablement notes
- System Specs x2, DFlash doc, 35B FP8 doc, Ling docs: stale active
  claims fixed, retired banners, never-shipped plan stubbed
This commit is contained in:
Frank committed 2026-08-17 03:30:00 -07:00
1 parent 3f88b8e1ed
commit 968066ff23
13 files changed
+189 -245
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

AGENTS.md
+15 -11
View File
@@ -15,11 +15,11 @@ The Dankclaw NPU endpoint (FastFlowLM, `frank-default-small`, port 52625)
serves Qwythos-9B-Claude-Mythos-5-1M (abliterated) as registry tag serves Qwythos-9B-Claude-Mythos-5-1M (abliterated) as registry tag
`qwythos:9b` at 262144 ctx (swapped in 2026-08-17; previously `qwythos:9b` at 262144 ctx (swapped in 2026-08-17; previously
`qwen3.6-moe:35b-a3b`; the old llama.cpp 35B service on Dankserver is `qwen3.6-moe:35b-a3b`; the old llama.cpp 35B service on Dankserver is
stopped). See [docs/deployment.md](docs/deployment.md) for the deployment stopped). **Boot policy (2026-08-17):** only `frank-default-large` and
process, [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md) for the `frank-pp-worker` are enabled for boot. The NPU unit `frank-default-small`
spec history and current production state, and is disabled; the running service keeps serving until stopped or rebooted.
[docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md](docs/Pipeline%20Parallelism/Qwen3.8%20PP%20Deployment.md) See [docs/deployment.md](docs/deployment.md) for the deployment
for the PP runbook. process, the boot policy, and the re-enable recipes.
**Sibling project:** [../serverAppConfigs](../serverAppConfigs) manages the **Sibling project:** [../serverAppConfigs](../serverAppConfigs) manages the
Podman app stack on the same hosts (OpenWebUI, Gitea, observability). Its Podman app stack on the same hosts (OpenWebUI, Gitea, observability). Its
@@ -83,10 +83,14 @@ Target map:
| Target | Host | Runtime | Service | Port | | Target | Host | Runtime | Service | Port |
|--------|------|---------|---------|------| |--------|------|---------|---------|------|
| Dankclaw | dank@dankclaw | llama.cpp PP head (Qwen3.8-27B UD-Q8_K_XL, split 0.40/0.60 to the Dankserver worker via RPC, MTP+ngram-mod, f16 KV) | frank-default-large | 7770 | | Dankclaw | dank@dankclaw | llama.cpp PP head (Qwen3.8-27B-Uncensored Q6_K, split 0.68/0.32 to the Dankserver worker via RPC, standalone MTP draft + ngram-mod, q8_0/f16 KV) | frank-default-large | 7770 |
| Dankclaw | dank@dankclaw | FastFlowLM NPU (XDNA2 50 TOPS, qwythos:9b Q4NX 7.76 GB, 262144 ctx; see [[FastFlowLM NPU Inference]]) | frank-default-small | 52625 | | Dankclaw | dank@dankclaw | FastFlowLM NPU (XDNA2 50 TOPS, qwythos:9b Q4NX 7.76 GB, 262144 ctx; see [[FastFlowLM NPU Inference]]) — disabled for boot since 2026-08-17 | frank-default-small | 52625 |
| Dankserver | dank@dankserver | ggml-rpc worker (RTX 3090, receives the head's tensor share) | frank-pp-worker | 50052 | | Dankserver | dank@dankserver | ggml-rpc worker (RTX 3090, receives the head's tensor share) | frank-pp-worker | 50052 |
> Warning: `deploy.sh dankserver` is currently broken. It targets the
> removed `frank-default-small` unit. Restart the worker with
> `sudo systemctl restart frank-pp-worker` until the script is fixed.
## Runtime & Troubleshooting ## Runtime & Troubleshooting
### Env var naming rule — HIP runtime collision ### Env var naming rule — HIP runtime collision
@@ -117,7 +121,7 @@ graph TD
2. Run the serve script directly. Then run the raw binary with the same flags. A different outcome isolates the env file sourcing. 2. Run the serve script directly. Then run the raw binary with the same flags. A different outcome isolates the env file sourcing.
3. Run gdb for the backtrace with this command: 3. Run gdb for the backtrace with this command:
``` ```
gdb -batch -ex run -ex "bt 25" --args bash scripts/serve-llama-frank-large.sh gdb -batch -ex run -ex "bt 25" --args bash scripts/serve-pp-head.sh
``` ```
`llama-server` buffers its timed stdout. On SIGSEGV the log tail is lost. Trust gdb, not the log. `llama-server` buffers its timed stdout. On SIGSEGV the log tail is lost. Trust gdb, not the log.
4. Kernel spam about `amdgpu: SVM mapping failed` is a symptom, not the root cause. Investigate other causes before assuming memory pressure. 4. Kernel spam about `amdgpu: SVM mapping failed` is a symptom, not the root cause. Investigate other causes before assuming memory pressure.
@@ -125,13 +129,13 @@ graph TD
### Operational quirks ### Operational quirks
- `sudo` on both boxes needs a tty. The remote sudo steps in `deploy.sh` fail non-interactively. Use `echo '<password>' | sudo -S <command>`. - `sudo` on both boxes needs a tty. The remote sudo steps in `deploy.sh` fail non-interactively. Use `echo '<password>' | sudo -S <command>`.
- Passwordless restarts: each target has a root-owned helper (`/usr/local/sbin/frank-restart-large.sh` on Dankclaw, `frank-restart-small.sh` on Dankserver) and a sudoers NOPASSWD drop-in (`/etc/sudoers.d/frank-*`). `deploy.sh` prefers the helper. The sudoers rules match commands exactly: no extra arguments (for example, `--no-pager` breaks the `systemctl status` rule). See [docs/deployment.md](docs/deployment.md). - Passwordless restarts: each target has a root-owned helper and a sudoers NOPASSWD drop-in (`/etc/sudoers.d/frank-*`). The Dankclaw helper `/usr/local/sbin/frank-restart-large.sh` is correct for the PP head. The Dankserver helper `frank-restart-small.sh` is **obsolete** — it restarts the removed `frank-default-small` unit. Restart the worker with `sudo systemctl restart frank-pp-worker`. The sudoers rules match commands exactly: no extra arguments (for example, `--no-pager` breaks the `systemctl status` rule). See [docs/deployment.md](docs/deployment.md).
- Never run `pkill -f` with a pattern that appears in your own command line. It kills your ssh session. Use `pkill -x <exact-name>`. - Never run `pkill -f` with a pattern that appears in your own command line. It kills your ssh session. Use `pkill -x <exact-name>`.
- Model load is slow: about 3 minutes warm, 8-17 minutes cold (measured 17 minutes on 2026-08-13 after a cache drop). Health returns `503 Loading model` the whole time. That is normal. Do not kill the process. `deploy.sh` waits up to 20 minutes on Dankclaw (see `HEALTH_WAIT`). - Model load is slow: about 3 minutes warm, 8-17 minutes cold (measured 17 minutes on 2026-08-13 after a cache drop). Health returns `503 Loading model` the whole time. That is normal. Do not kill the process. `deploy.sh` waits up to 20 minutes on Dankclaw (see `HEALTH_WAIT`).
- Page-cache state confounds crash tests. Cold-cache loads are slow and survive. Warm-cache loads crash fast. Control for it when testing. `deploy.sh` drops caches on Dankclaw before restart (see `DROP_CACHES`); manual restarts must run `sync && echo 3 | sudo tee /proc/sys/vm/drop_caches` first. - Page-cache state confounds crash tests. Cold-cache loads are slow and survive. Warm-cache loads crash fast. Control for it when testing. `deploy.sh` drops caches on Dankclaw before restart (see `DROP_CACHES`); manual restarts must run `sync && echo 3 | sudo tee /proc/sys/vm/drop_caches` first.
- A client abort and a server release look different in the journal. A client abort logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in [docs/deployment.md](docs/deployment.md). - A client abort and a server release look different in the journal. A client abort logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in [docs/deployment.md](docs/deployment.md).
- Dankserver crash signature under parallel load: `CUDA error: an unsupported value or parameter was passed to the function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541`, then `status=6/ABRT`, then `503 Loading model` for ~5 minutes. Happened twice on 2026-08-13 under a 5-researcher subagent storm (llama.cpp b10121, driver 595.84). Root cause not yet determined; do not confuse it with speculative decoding. - Dankserver crash signature (historical): the retired 35B service aborted under parallel load with `CUDA error: an unsupported value or parameter was passed to the function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541`, then `status=6/ABRT`, then `503 Loading model` for ~5 minutes (llama.cpp b10121, driver 595.84). Root cause was never determined; the service is retired. Do not confuse that crash with speculative decoding. See [docs/dankserver-vram-growth.md](docs/dankserver-vram-growth.md).
- DFlash premature EOS under parallel load: on Dankclaw, `draft-dflash` ends a generating slot ~1s after a new task lands with high prompt-cache reuse (LCP ~0.99), with `finish_reason "stop"` and no `cancel task` log. It is a DFlash shared-draft-context bug. DFlash is retired from Dankclaw. DSpark (`draft-dspark`, its semi-autoregressive successor) is also unsafe here: two concurrent 16k prose prompts crashed the ROCm backend with `ROCm error: an illegal memory access` (status 6/ABRT). Production uses `--spec-default` (ngram-mod, draftless) at PARALLEL=2 on the Vulkan0 backend. See [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md). - DFlash premature EOS under parallel load (historical): on Dankclaw, `draft-dflash` ended a generating slot ~1s after a new task landed with high prompt-cache reuse (LCP ~0.99), with `finish_reason "stop"` and no `cancel task` log — a DFlash shared-draft-context bug. DFlash and DSpark are retired from Dankclaw. Production uses the standalone MTP draft over the ngram-mod fallback at PARALLEL=1 on ROCm0. See [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md).
### Dankclaw memory model ### Dankclaw memory model
README.md
+22 -18
View File
@@ -1,11 +1,13 @@
# vLLM Scripts # vLLM Scripts
Scripts for running LLM inference on two local machines. Deployment scripts for the LLM inference fleet: a llama.cpp
pipeline-parallel Qwen3.8-27B split across two machines, plus the
FastFlowLM NPU endpoint on Dankclaw.
## Targets ## Targets
| Host | User | Codename | Architecture | | Host | User | Codename | Architecture |
|------|------|----------|-------------| |------|------|----------|--------------|
| dankserver | dank | Dankserver | NVIDIA RTX 3090 + 62 GB RAM | | dankserver | dank | Dankserver | NVIDIA RTX 3090 + 62 GB RAM |
| dankclaw | dank | Dankclaw | AMD Strix Halo APU (shared memory) | | dankclaw | dank | Dankclaw | AMD Strix Halo APU (shared memory) |
@@ -13,29 +15,31 @@ Scripts for running LLM inference on two local machines.
| Service | Host | Runtime | Model | Port | | Service | Host | Runtime | Model | Port |
|---------|------|---------|-------|------| |---------|------|---------|-------|------|
| frank-default-large | dankclaw | vLLM-ROCm (CIRU) | Ling-3.0-flash CIRU INT4 MTP | 7770 | | frank-default-large | dankclaw | llama.cpp PP head (RPC to Dankserver) | Qwen3.8-27B-Uncensored Q6_K, standalone MTP draft | 7770 |
| frank-default-small | dankserver | llama.cpp (ngram-mod) | Qwen3.6-35B-A3B uncensored Q6_K_P | 7770 | | frank-pp-worker | dankserver | ggml-rpc worker (RTX 3090) | receives the head's tensor share | 50052 |
> The retired ROCmFPX llama.cpp service on Dankclaw was replaced by the vLLM service above. The retired `vllm-awq-qwen36` (AWQ 4-bit) service on Dankserver was replaced by the llama.cpp service above. Legacy unit files remain in `scripts/` for reference. Boot policy (2026-08-17): only these two units are enabled. The
FastFlowLM NPU endpoint (`frank-default-small`, qwythos:9b, port 52625)
## Models is disabled for boot; re-enable it from `scripts/frank-default-small.service`
if needed. Retired units: the Dankserver llama.cpp 35B service,
| Model | Target | Format | `llamaserver`, `ollama`, `vllm-awq-qwen36`, `vllm-qwen36`,
|-------|--------|--------| `frank-large-ling` (Ling trial), and the vLLM-ROCm CIRU path. Their unit
| Ling-3.0-flash CIRU INT4 (MTP) | dankclaw | INT4 W4A16 (Quark/vLLM) | files remain in `scripts/` for reference.
| Qwen3.6 35B A3B uncensored Q6_K_P (ngram-mod speculative decoding) | dankserver | GGUF Q6_K_P (llama.cpp, MoE offload) |
| Qwen3.6 35B A3B AWQ 4bit | dankserver | AWQ (retired) |
| Qwen3.6 35B A3B FP8 | dankserver | FP8 (documented, not active) |
| Qwen3.6 27B FP8 | dankclaw | FP8 (documented, not active) |
## Structure ## Structure
- `docs/` — Project knowledge base with research notes on vLLM configuration, hardware specifics, speculative decoding, and deployment notes - `docs/` — knowledge base: deployment runbook, PP runbook, spec matrix,
- `scripts/` — Deployment and launch scripts (deploy.sh, serve-llama-frank-small.sh, serve-ling3-vllm.sh, systemd unit files) NPU notes, research archive
- `scripts/` — serve scripts, env files, systemd units, deploy and test
tools
## Deployment ## Deployment
See [docs/deployment.md](docs/deployment.md) for the deploy/update process. Run `./scripts/deploy.sh <dankclaw|dankserver>` after pushing committed changes. See [docs/deployment.md](docs/deployment.md). Run
`./scripts/deploy.sh dankclaw` for the PP head. The `dankserver` target
is broken (it targets the removed `frank-default-small` unit); restart
the worker with `sudo systemctl restart frank-pp-worker` until the
script is fixed.
## Knowledge Base ## Knowledge Base
docs/DFlash DSpark Speculative Decoding.md
+2 -12
View File
@@ -160,9 +160,7 @@ The DFlash draft is not a standalone model. It must pair with its target model.
## Deployment on Dankserver (NVIDIA RTX 3090) ## Deployment on Dankserver (NVIDIA RTX 3090)
## Deployment on Dankserver (NVIDIA RTX 3090) The Dankserver llama.cpp deployment (`frank-default-small`, Qwen3.6-35B-A3B) is retired (2026-08-16) and its unit disabled (2026-08-17). The RTX 3090 now runs the PP worker for the Dankclaw head (see [[Qwen3.8 PP Deployment]]). The vLLM material below documents the archived AWQ/DFlash path on the same hardware.
The active Dankserver deployment runs **llama.cpp** (`frank-default-small`), not vLLM. It serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` on port 7770 using **ngram-mod** speculative decoding (n_min=48 / n_max=64 / n_match=24) with 3 slots at 200000 context each and MoE experts on CPU. See [[deployment]] and `scripts/serve-llama-frank-small.sh`.
> **DFlash retired from active service.** `DFlash` (draft-dflash) was tested initially but did not deliver the desired speed gains under the `frank-default-small` workload (see ngram-mod benchmarks below). The `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` draft GGUF remains in `/home/dank/models/` as reference material only. > **DFlash retired from active service.** `DFlash` (draft-dflash) was tested initially but did not deliver the desired speed gains under the `frank-default-small` workload (see ngram-mod benchmarks below). The `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` draft GGUF remains in `/home/dank/models/` as reference material only.
@@ -197,15 +195,7 @@ docker run --rm --name vllm-qwen36-dflash \
## Deployment on Dankclaw (AMD Strix Halo gfx1151) ## Deployment on Dankclaw (AMD Strix Halo gfx1151)
Dankclaw now runs llama.cpp as its runtime for `frank-default-large` (port 7770), serving the unsloth Qwen3.8-27B `UD-Q8_K_XL` model with **no dedicated draft** on the **Vulkan0** backend. `frank-default-large` on Dankclaw runs the llama.cpp pipeline-parallel head (Qwen3.8-27B-Uncensored Q6_K) on port 7770, RPC-split with the Dankserver worker. Spec is the standalone MTP draft over the ngram-mod fallback, 1 slot at 262144 ctx, q8_0/f16 KV, head GPU side ROCm0. See [[qwen3.8-spec-matrix]] and [[Qwen3.8 PP Deployment]] for the current state and the 2026-08-15 Vulkan0/ROCm0 high-context bench.
Current facts (2026-08-15):
- Main model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (bundled MTP, unused).
- Spec: llama.cpp **`--spec-default`** preset = ngram-mod (n_match 24 / n_min 48 / n_max 64). No draft model.
- **Backend: Vulkan0** — at 262144 ctx (~200k prompt) Vulkan decodes 5.68 t/s vs ROCm 3.83 t/s (48% faster); ROCm wins prefill 119 vs 106 t/s. See [[qwen3.8-spec-matrix]] for the bench table.
- **PARALLEL=2** — passes the 2x32k parallel stream test.
- KV `q8_0` K / `q8_0` V; context 2 x 262144; batch 8192 / ubatch 256.
History: before `--spec-default`, the box ran DSpark (`draft-dspark`, retired 2026-08-15 after a ROCm illegal-memory-access crash under two concurrent 16k prompts) and before that bundled MTP (`draft-mtp`, retired same day). DFlash was retired on 2026-08-14 (premature-EOS under parallel load). The old DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`), the MTP-only drafter (`Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`), and the DSpark draft (`Qwen3.8-27B-DSpark-Q8_0.gguf`) remain in `/home/dank/models/` as reference only. History: before `--spec-default`, the box ran DSpark (`draft-dspark`, retired 2026-08-15 after a ROCm illegal-memory-access crash under two concurrent 16k prompts) and before that bundled MTP (`draft-mtp`, retired same day). DFlash was retired on 2026-08-14 (premature-EOS under parallel load). The old DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`), the MTP-only drafter (`Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`), and the DSpark draft (`Qwen3.8-27B-DSpark-Q8_0.gguf`) remain in `/home/dank/models/` as reference only.
docs/Dankclaw System Specs.md
+1 -1
View File
@@ -82,7 +82,7 @@ The NPU shares the 30 GiB unified pool. FLM allocates roughly half of system RAM
- ROCm reports 96 GiB of GPU-visible memory (overcommitted). The current vLLM service runs at `--gpu-memory-utilization 0.95`, which fits the CIRU INT4 weights and 256k KV cache (about 90 GiB used of the 96 GiB frame buffer). - ROCm reports 96 GiB of GPU-visible memory (overcommitted). The current vLLM service runs at `--gpu-memory-utilization 0.95`, which fits the CIRU INT4 weights and 256k KV cache (about 90 GiB used of the 96 GiB frame buffer).
- Without the ROCm VRAM patch, ROCm clamps APU memory to 50% (see `scripts/patches/rocm_patched.py` on the remote). The patch restores the real GTT ceiling minus an 8 GB safety margin. - Without the ROCm VRAM patch, ROCm clamps APU memory to 50% (see `scripts/patches/rocm_patched.py` on the remote). The patch restores the real GTT ceiling minus an 8 GB safety margin.
- Only 8 GiB swap. Modern vLLM uses `--kv-offloading-size` instead of the removed `--swap-space` flag. - Only 8 GiB swap. Modern vLLM uses `--kv-offloading-size` instead of the removed `--swap-space` flag.
- The active service on Dankclaw is `frank-default-large.service`: vLLM-ROCm serving the Ling-3.0-flash CIRU INT4 (W4A16) model on port 7770 with native MTP speculative decoding. It replaced the ROCmFPX llama.cpp service. - The active service on Dankclaw is `frank-default-large.service`: the llama.cpp pipeline-parallel head (Qwen3.8-27B-Uncensored Q6_K) on port 7770, RPC-split with the Dankserver worker. The vLLM-ROCm CIRU Ling path was planned but never shipped. See [[deployment]].
- ROCm 7.2.4 from AMD repos. The CIRU venv bundles a self-contained ROCm 7.15 toolchain; vLLM ROCm support requires ROCm 6.3+. - ROCm 7.2.4 from AMD repos. The CIRU venv bundles a self-contained ROCm 7.15 toolchain; vLLM ROCm support requires ROCm 6.3+.
- Python 3.14.4 is very new. vLLM ROCm wheels target Python 3.12. The CIRU venv uses Python 3.12. - Python 3.14.4 is very new. vLLM ROCm wheels target Python 3.12. The CIRU venv uses Python 3.12.
- Radeon 8060S falls under Ryzen AI MAX series (gfx1151). vLLM lists this as supported. - Radeon 8060S falls under Ryzen AI MAX series (gfx1151). vLLM lists this as supported.
docs/Dankserver System Specs.md
+2 -2
View File
@@ -51,11 +51,11 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 24.0
### Active Services ### Active Services
The active `frank-default-small` service (llama.cpp, `b10108+`) serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model on port 7770 with **ngram-mod** speculative decoding. It runs **4 parallel slots** at 175104 context each (pool 700416, refitted 2026-08-14 for the dual-f16 KV cache). Of the 40 layers' MoE expert weights, **36 sit on CPU** (`--n-cpu-moe 36`) while the remaining **4 layers' expert weights** are resident on GPU along with all non-expert tensors (`--gpu-layers all`). KV stays on GPU with K=`f16`, V=`f16` (13680 MiB pool at 700416 cells). Batch 8192 / ubatch 3072, `--ctx-checkpoints 4`, `--no-mmap`. At idle ~22.1 GB VRAM used, leaving ~2.0 GiB headroom. Idle RAM ~32 GB of 62 GB. The active service on Dankserver is `frank-pp-worker.service`: the ggml-rpc worker (RTX 3090) that receives the PP head's tensor share (45 of 65 blocks, ~23.8 GiB of 24 GiB at steady state). The old llama.cpp Qwen3.6-35B-A3B service (`frank-default-small`) is retired (2026-08-16) and disabled (2026-08-17). See [[Qwen3.8 PP Deployment]].
## Hardware Relevance ## Hardware Relevance
- 24 GB VRAM is close to saturated with this config. At idle ~22.1 GB used, leaving ~2.0 GiB headroom at `--n-cpu-moe 36` (4 layers' experts on GPU) with `ubatch 3072` and a dual-f16 KV pool of 13680 MiB at 700416 cells (2026-08-14 refit; the same pool at 800768 cells OOM'd the load warmup). A llama-bench sweep ([[frank-default-small-benchmark]]) guided this state: prefill is the priority, so `ubatch 3072` stays (top prefill tier), and `ncmoe 36` was raised from 35 to fix a cuBLAS OOM-class crash during big prefills by freeing ~0.75 GiB peak VRAM at only -2% prefill. If it still crashes under load, add `GGML_CUDA_FORCE_MMQ=1`. If decode speed matters more, use `ncmoe 32 - ubatch 1024` (gen ~48.7 t/s). By editing `scripts/frank-default-small.env`. - 24 GB VRAM is close to saturated with this config. At idle ~22.1 GB used, leaving ~2.0 GiB headroom at `--n-cpu-moe 36` (4 layers' experts on GPU) with `ubatch 3072` and a dual-f16 KV pool of 13680 MiB at 700416 cells (2026-08-14 refit; the same pool at 800768 cells OOM'd the load warmup). A llama-bench sweep ([[frank-default-small-benchmark]]) guided this state: prefill is the priority, so `ubatch 3072` stays (top prefill tier), and `ncmoe 36` was raised from 35 to fix a cuBLAS OOM-class crash during big prefills by freeing ~0.75 GiB peak VRAM at only -2% prefill. If it still crashes under load, add `GGML_CUDA_FORCE_MMQ=1`. If decode speed matters more, use `ncmoe 32 - ubatch 1024` (gen ~48.7 t/s). The tuning guidance applied to the retired `frank-default-small` service; the PP worker's allocations are fixed at load.
- 62 GB system RAM holds the ~29 GB of MoE expert weights with room to spare (~57 GB available after load). - 62 GB system RAM holds the ~29 GB of MoE expert weights with room to spare (~57 GB available after load).
- 127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled. - 127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled.
- Root disk has approximately 140 GB free. Store the GGUF models under `/home/dank/models` (about 32 GB total). - Root disk has approximately 140 GB free. Store the GGUF models under `/home/dank/models` (about 32 GB total).
docs/FastFlowLM NPU Inference.md
+10
View File
@@ -200,6 +200,16 @@ Notes:
`frank-default-small` now runs the NPU endpoint on Dankclaw (was the Dankserver llama.cpp service). Unit: `scripts/frank-default-small.service` (symlinked into `/etc/systemd/system`), wrapper `scripts/serve-flm-small.sh`, tunables `scripts/frank-default-small.env`. `LimitMEMLOCK=infinity` is set in the unit. Default model: `gemma4-it:e4b`. Swap `FLM_MODEL` in the env to switch. `frank-default-small` now runs the NPU endpoint on Dankclaw (was the Dankserver llama.cpp service). Unit: `scripts/frank-default-small.service` (symlinked into `/etc/systemd/system`), wrapper `scripts/serve-flm-small.sh`, tunables `scripts/frank-default-small.env`. `LimitMEMLOCK=infinity` is set in the unit. Default model: `gemma4-it:e4b`. Swap `FLM_MODEL` in the env to switch.
### Boot policy (2026-08-17)
The `frank-default-small` unit is disabled for boot. Only the PP pair (`frank-default-large` + `frank-pp-worker`) starts at boot. The unit file was removed from `/etc/systemd/system`; the running FLM service keeps serving until stopped or rebooted. To restore the endpoint, run these commands on Dankclaw:
```
sudo ln -sf /home/dank/vllmScripts/scripts/frank-default-small.service /etc/systemd/system/frank-default-small.service
sudo systemctl daemon-reload
sudo systemctl enable frank-default-small
```
Live config (2026-08-16): `FLM_PMODE=turbo`, `FLM_ASR=1` (loads Whisper-V3-Turbo-NPU2; exposes `/v1/audio/transcriptions`), `FLM_CTX_LEN=131072` (verified max — Gemma4 natively supports 128K and the NPU budget fits it; KV occupancy measured at 0.11% for a 123-token exchange, decode 12.0 t/s, 13 GiB headroom). Live config (2026-08-16): `FLM_PMODE=turbo`, `FLM_ASR=1` (loads Whisper-V3-Turbo-NPU2; exposes `/v1/audio/transcriptions`), `FLM_CTX_LEN=131072` (verified max — Gemma4 natively supports 128K and the NPU budget fits it; KV occupancy measured at 0.11% for a 123-token exchange, decode 12.0 t/s, 13 GiB headroom).
The PP split was renamed `frank-pp-head` → `frank-default-large` (2026-08-16, commit `48142f5`). The PP split was renamed `frank-pp-head` → `frank-default-large` (2026-08-16, commit `48142f5`).
docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md
+14 -1
View File
@@ -1,6 +1,6 @@
# Qwen3.8-27B Pipeline-Parallel Deployment (Dankclaw + Dankserver) # Qwen3.8-27B Pipeline-Parallel Deployment (Dankclaw + Dankserver)
Split Qwen3.8-27B UD-Q8_K_XL across the Strix Halo iGPU (head) and the RTX Split Qwen3.8-27B across the Strix Halo iGPU (head) and the RTX
3090 (worker) using llama.cpp RPC. This page is the design record and the 3090 (worker) using llama.cpp RPC. This page is the design record and the
morning-of runbook. Research background: morning-of runbook. Research background:
[Qwen3.8 Strix optimizations](../../qwen3.8-strix-optimizations.md). [Qwen3.8 Strix optimizations](../../qwen3.8-strix-optimizations.md).
@@ -31,6 +31,19 @@ state, verified against the live journal and output smoke tests
- Unit `frank-default-large.service`, launched through - Unit `frank-default-large.service`, launched through
`serve-pp-head.sh` + `frank-default-large.env`. `serve-pp-head.sh` + `frank-default-large.env`.
## Service enablement (2026-08-17)
The PP pair is the only inference service enabled for boot. Both units
(`frank-default-large` on Dankclaw, `frank-pp-worker` on Dankserver) are
enabled. The NPU unit (`frank-default-small` on Dankclaw) and all retired
units are disabled. Restart order: worker first, then head. The Dankserver
helper `frank-restart-small.sh` is obsolete (it restarts the removed
`frank-default-small` unit); restart the worker with
`sudo systemctl restart frank-pp-worker`. The Dankclaw helper
`frank-restart-large.sh` (sync + cache drop + restart) is correct for the
head. `deploy.sh dankserver` is broken until retargeted. See
[[deployment]] for the full boot-policy table.
## FA-all-quants rebuild (2026-08-17) ## FA-all-quants rebuild (2026-08-17)
Flash Attention with mismatched K/V cache types requires the Flash Attention with mismatched K/V cache types requires the
docs/Running Qwen3.6 35B A3B FP8 on Dankserver.md
+3 -2
View File
@@ -1,5 +1,7 @@
# Running Qwen3.6 35B A3B FP8 on Dankserver # Running Qwen3.6 35B A3B FP8 on Dankserver
> **RETIRED.** This page documents the FP8 vLLM path for Qwen3.6-35B-A3B on Dankserver. The llama.cpp 35B service is retired (2026-08-16), its unit file was removed (2026-08-17), and the FP8 vLLM unit (`vllm-qwen36`) never shipped and is disabled. The content is kept as research history. The RTX 3090 now runs the PP worker: see [[Qwen3.8 PP Deployment]].
## Model Information ## Model Information
Qwen3.6-35B-A3B is a Mixture-of-Experts vision-language model from the Qwen3.6 series. It uses 35B total parameters with only 3B activated per token. Qwen3.6-35B-A3B is a Mixture-of-Experts vision-language model from the Qwen3.6 series. It uses 35B total parameters with only 3B activated per token.
@@ -112,10 +114,9 @@ export HF_HOME=/mnt/entropy/huggingface
### Active Service Conflict ### Active Service Conflict
The active `frank-default-small` service (llama.cpp, port 7770) uses about 22 GB of the 24 GB VRAM. The retired `vllm-awq-qwen36` (Podman, ~22 GB) and `vllm-qwen36` services are not active. Stop any competing service before starting another: The retired `frank-default-small` service (llama.cpp) used about 22 GB of the 24 GB VRAM. The current PP worker holds ~23.8 GiB. The `vllm-awq-qwen36` (Podman) and `vllm-qwen36` units are disabled. Before starting any research service, verify that the GPU is free:
``` ```
podman stop vllm-qwen36-dflash
nvidia-smi # verify GPU is free nvidia-smi # verify GPU is free
``` ```
docs/deployment.md
+53 -26
View File
@@ -8,11 +8,43 @@ Two servers run LLM inference services. Each server pulls its scripts from this
| Target | Host | Runtime | Service | Port | | Target | Host | Runtime | Service | Port |
|--------|------|---------|---------|------| |--------|------|---------|---------|------|
| Dankclaw | dank@dankclaw | llama.cpp PP head (Qwen3.8-27B UD-Q8_K_XL) | frank-default-large | 7770 | | Dankclaw | dank@dankclaw | llama.cpp PP head (Qwen3.8-27B-Uncensored Q6_K, split 0.68 to the Dankserver worker) | frank-default-large | 7770 |
| Dankclaw | dank@dankclaw | FastFlowLM NPU (qwythos:9b, 262144 ctx) | frank-default-small | 52625 | | Dankclaw | dank@dankclaw | FastFlowLM NPU (qwythos:9b, 262144 ctx) | frank-default-small | 52625 |
| Dankserver | dank@dankserver | ggml-rpc worker (RTX 3090) | frank-pp-worker | 50052 | | Dankserver | dank@dankserver | ggml-rpc worker (RTX 3090) | frank-pp-worker | 50052 |
The `frank-default-small` (NPU) unit is not in `deploy.sh`. Restart it by hand with `sudo systemctl restart frank-default-small`. The `frank-default-small` (NPU) unit is not in `deploy.sh` and is disabled for boot (see Boot Policy below). If the unit is still loaded (pre-reboot), restart it by hand with `sudo systemctl restart frank-default-small`. After a reboot, re-enable the unit first (see Boot Policy).
## Boot Policy (2026-08-17)
Only the pipeline-parallel pair starts at boot: `frank-default-large` (Dankclaw head) and `frank-pp-worker` (Dankserver worker). Every other inference unit is disabled. This was set on 2026-08-17.
Disabled units:
| Machine | Unit | Fate |
|---------|------|------|
| Dankclaw | `frank-default-small` (FastFlowLM NPU, port 52625) | Disabled; unit file removed from `/etc/systemd/system`. The service keeps running until stopped or rebooted. |
| Dankclaw | `frank-large-ling` (Ling trial) | Disabled. |
| Dankclaw | `vllm-thinkingcap` | Disabled (already). |
| Dankserver | `frank-default-small` (old llama.cpp 35B) | Disabled; unit file removed. |
| Dankserver | `llamaserver`, `ollama`, `vllm-awq-qwen36`, `vllm-qwen36` | Disabled. |
`systemctl disable` removes the symlinked unit file from `/etc/systemd/system` when the file is a symlink. The repo copies stay intact. To restore a disabled service, re-symlink the unit and enable it:
```
sudo ln -sf /home/dank/<repo>/scripts/<unit>.service /etc/systemd/system/<unit>.service
sudo systemctl daemon-reload
sudo systemctl enable <unit>
```
The NPU re-enable recipe uses the Dankclaw repo path `/home/dank/vllmScripts/scripts/frank-default-small.service`.
The Dankserver helper `frank-restart-small.sh` targets the removed `frank-default-small` unit. Do not use it. Restart the PP worker with an explicit command:
```
sudo systemctl restart frank-pp-worker
```
`scripts/deploy.sh dankserver` also targets the removed unit. Do not run it until the script is retargeted to `frank-pp-worker`. The `dankclaw` target works: it restarts the PP head through `frank-restart-large.sh`.
## Workflow ## Workflow
@@ -77,7 +109,7 @@ Each target has a root-owned helper script that performs the full restart. The u
| Target | Helper | Sudoers drop-in | | Target | Helper | Sudoers drop-in |
|--------|--------|-----------------| |--------|--------|-----------------|
| Dankclaw | `/usr/local/sbin/frank-restart-large.sh` | `/etc/sudoers.d/frank-large` | | Dankclaw | `/usr/local/sbin/frank-restart-large.sh` | `/etc/sudoers.d/frank-large` |
| Dankserver | `/usr/local/sbin/frank-restart-small.sh` | `/etc/sudoers.d/frank-small` | | Dankserver | `/usr/local/sbin/frank-restart-small.sh` (obsolete, see below) | `/etc/sudoers.d/frank-small` |
The Dankclaw helper does these steps: The Dankclaw helper does these steps:
@@ -87,7 +119,7 @@ The Dankclaw helper does these steps:
4. Reloads systemd. 4. Reloads systemd.
5. Enables and restarts the service. 5. Enables and restarts the service.
The Dankserver helper does the same steps without the cache drop. The Dankserver helper `frank-restart-small.sh` is obsolete: it restarts the removed `frank-default-small` unit. Do not use it. The sudoers drop-in does not cover `frank-pp-worker`, so the worker restart needs the password (or a new helper). Restart the worker with `sudo systemctl restart frank-pp-worker`.
The sudoers drop-in contains exactly two rules: The sudoers drop-in contains exactly two rules:
@@ -125,17 +157,24 @@ curl -sf http://127.0.0.1:7770/health
### Update Dankserver ### Update Dankserver
Run these commands on Dankserver: Run these commands on Dankserver to update and restart the PP worker:
``` ```
cd /home/dank/vLLM cd /home/dank/vLLM
git pull origin main git pull origin main
sudo ln -sf /home/dank/vLLM/scripts/frank-default-small.service /etc/systemd/system/frank-default-small.service sudo ln -sf /home/dank/vLLM/scripts/frank-pp-worker.service /etc/systemd/system/frank-pp-worker.service
sudo systemctl daemon-reload sudo systemctl daemon-reload
sudo systemctl restart frank-default-small sudo systemctl restart frank-pp-worker
curl -sf http://127.0.0.1:7770/health
``` ```
The worker has no HTTP health endpoint. Verify the listener instead:
```
ss -tln | grep :50052
```
The old passwordless helper `frank-restart-small.sh` targets the removed `frank-default-small` unit. Do not use it. See Boot Policy.
## Service Units ## Service Units
### Dankclaw — frank-default-large ### Dankclaw — frank-default-large
@@ -189,15 +228,9 @@ layer offload (`--gpu-layers all`, `--fit off`), batch 8192 / ubatch
> table. Also retired earlier: the vLLM-ROCm CIRU INT4 Ling-3.0-flash > table. Also retired earlier: the vLLM-ROCm CIRU INT4 Ling-3.0-flash
> runtime, and the llama-serve/vLLM units on Dankclaw. > runtime, and the llama-serve/vLLM units on Dankclaw.
### Dankserver — frank-default-small (STOPPED 2026-08-16) ### Dankserver — old llama.cpp 35B service (retired 2026-08-16)
> Note (2026-08-16): the `frank-default-small` alias now belongs to the FastFlowLM NPU endpoint on Dankclaw (port 52625, see [[FastFlowLM NPU Inference]]). The llama.cpp Qwen3.6-35B-A3B service on Dankserver described below is stopped (verified `inactive` 2026-08-16). The text below is kept as history. Dankserver previously ran `frank-default-small`: llama.cpp serving the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` (ngram-mod spec, 4 slots x 175104 ctx, 36 of 40 MoE expert layers on CPU, mmproj vision). The service was verified `inactive` on 2026-08-16, and its unit file was removed on 2026-08-17 under the boot policy. The name `frank-default-small` now belongs to the FastFlowLM NPU endpoint on Dankclaw (see [[FastFlowLM NPU Inference]]). The RTX 3090 now runs the PP worker. See git history for the full retired configuration.
The active service on Dankserver runs llama.cpp (upstream, `b10108+`) and serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model under the alias `frank-default-small` on port 7770. It uses **ngram-mod** speculative decoding (`--spec-ngram-mod-n-min 48 / n_max 64 / n_match 24`), **4 parallel slots** at 175104 context each (pool 700416), and mmproj vision. **36 of 40 MoE expert layers** stay on CPU (`--n-cpu-moe 36`); 4 expert layers' weights are back on GPU alongside all non-expert tensors (`--gpu-layers all`). KV stays on GPU (K=`f16`, V=`f16`), refitted from 800000 on 2026-08-14 when the pool went dual-f16 (the f16/f16 pool costs 13680 MiB at 700416 cells and did not fit at 800768 cells). Batch 8192 / ubatch 3072, `--ctx-checkpoints 4`, `--no-mmap`. The unit file is `scripts/frank-default-small.service`, symlinked to `/etc/systemd/system/`. It launches through `scripts/serve-llama-frank-small.sh`, which reads tunables from `scripts/frank-default-small.env`.
> Note: the Dankserver process currently observed runs `--spec-type ngram-mod` with `--parallel 4`, batch 8192 / ubatch 2048, and `--n-cpu-moe 34` (this is being bench-tuned by the operator; reconcile against `scripts/frank-default-small.env`).
> Note: this service replaced the retired `vllm-awq-qwen36` (AWQ 4-bit, port 34401) and the earlier `vllm-qwen36` (FP8) / `llama-server` (ROCmFPX) units. The repo tracks `main` on Dankserver at `/home/dank/vLLM`.
## Validation ## Validation
@@ -230,17 +263,11 @@ If a service segfaults at load with no obvious cause, compare the exported env a
## Incident: Jobs "Released Prematurely" Under Parallel Load (2026-08-13) ## Incident: Jobs "Released Prematurely" Under Parallel Load (2026-08-13)
The user reported that on `frank-default-large` (Dankclaw), a job that is streaming tokens gets released early when another job starts streaming. The suspicion was the speculative decoding configuration ("dspark"). Investigation found no dspark in either deployment: Dankclaw runs `draft-dflash`, Dankserver runs `ngram-mod`. Three findings emerged: A user reported that a streaming job on `frank-default-large` was released early when another job started. The suspicion was speculative decoding. The investigation found three facts:
1. **The server does not preempt.** `llama-server` defers a new task into a queue when all slots are busy. It never releases a running generation to make room. Verified empirically: two concurrent streaming jobs completed normally with dissimilar prompts, similar prompts, and near-identical prompts (high LCP reuse). `parallel=3` on Dankclaw serves up to three concurrent jobs and queues the rest. 1. **The server does not preempt.** `llama-server` defers a new task into a queue when all slots are busy. It never releases a running generation to make room. Verified empirically with concurrent streaming jobs.
2. **The releases were client-side aborts.** The pi coding agent aborts a stream when the user interrupts it (for example, Escape, or starting a new message in a way that cancels the previous one). The client records `stopReason: "aborted"` with `errorMessage: "Operation aborted"`, and the server logs `srv stop: cancel task, id_task = N`. The server journal shows a `slot release` line with no `cancel task` line when the generation ends normally (EOS or limit). 2. **The releases were client-side aborts.** The pi coding agent aborts a stream when the user interrupts it. The client records `stopReason: "aborted"`; the server logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`.
3. **Dankserver crashed under the parallel researcher load.** Five parallel research subagents hammered `frank-default-small` with back-to-back requests (a new task every few seconds for ~15 minutes). At 19:45:44 and again at 19:55:27 the service aborted with `CUDA error: an unsupported value or parameter was passed to the function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541`. systemd restarted it, and in-flight streams died with `Connection error.` or `terminated`, then `503 Loading model` during the ~5 minute reload. This crash, not speculative decoding, killed the parallel jobs. The signature is a CUBLAS invalid-value error in a GEMM under heavy concurrent load; root cause on Dankserver (llama.cpp b10121, driver 595.84, MoE experts CPU-offloaded) was not yet determined. 3. **Dankserver crashed under the parallel researcher load.** Five parallel research subagents hammered `frank-default-small` with back-to-back requests. The service aborted twice with `CUDA error: an unsupported value or parameter was passed to the function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541`, then `status=6/ABRT`, then `503 Loading model` for ~5 minutes. That crash, not speculative decoding, killed the parallel jobs. The affected service is retired; see [[dankserver-vram-growth]] for the VRAM/RAM growth findings behind the crash class.
Diagnosis recipe: to tell a client abort from a server release, read the journal around the release time. A client abort shows `W srv stop: cancel task, id_task = N`. A natural stop shows only `slot release` with `truncated = 0` and the final timings line.
### VRAM and RAM growth under load
This crash class recurs on Dankserver. The 2026-08-14 investigation ruled out ctx-checkpoints and ngram-mod as VRAM growth sources and confirmed RAM growth. See [[dankserver-vram-growth]] for the full findings, the monitoring setup, and the mitigation options.
## See Also ## See Also
docs/index.md
+8 -6
View File
@@ -19,21 +19,23 @@ frank-pp-worker, port 50052]
D[gfx1151 APU 30GiB phys / 96GiB GPU-visible] D[gfx1151 APU 30GiB phys / 96GiB GPU-visible]
E[llama.cpp PP head E[llama.cpp PP head
frank-default-large, port 7770] frank-default-large, port 7770]
F[Qwen3.8-27B UD-Q8_K_XL F[Qwen3.8-27B-Uncensored Q6_K
split 0.40/0.60 to worker, MTP+ngram-mod] split 0.68/0.32 to worker, standalone MTP draft + ngram-mod]
D --> E --> F D --> E --> F
G[FastFlowLM NPU XDNA2 50 TOPS G[FastFlowLM NPU XDNA2 50 TOPS
frank-default-small, port 52625] frank-default-small, port 52625 - DISABLED for boot 2026-08-17]
H[qwen3.6-moe:35b-a3b H[qwythos:9b
12.5-13.1 t/s, 262k ctx, vision] Q4NX 7.76 GB, 262k ctx]
D --> G --> H D --> G --> H
end end
I[LLM Clients / API] -->|HTTP :7770| E I[LLM Clients / API] -->|HTTP :7770| E
I -->|HTTP :52625| G I -->|HTTP :52625| G
E -->|RPC layers 0.40| B E -->|RPC layers 0.68| B
``` ```
Boot policy (2026-08-17): only `frank-default-large` and `frank-pp-worker` are enabled. The NPU endpoint and all retired units are disabled; see [[deployment]] for the table and re-enable recipes.
## Servers ## Servers
| Server | GPU | VRAM / Memory | OS | | Server | GPU | VRAM / Memory | OS |
docs/ling-3.0-flash-trial.md
+8 -11
View File
@@ -1,5 +1,7 @@
# Ling-3.0-flash Q6_K Trial on Dankclaw # Ling-3.0-flash Q6_K Trial on Dankclaw
> **RETIRED 2026-08-17.** The `frank-large-ling` unit is disabled under the boot policy. `frank-default-large` now runs the Qwen3.8-27B PP head, not the Qwen3.6-27B config described in the (obsolete) switch-back section below.
This page records the Ling-3.0-flash trial that temporarily replaced This page records the Ling-3.0-flash trial that temporarily replaced
`frank-default-large` on Dankclaw. It replaces the Qwen3.6-27B service on `frank-default-large` on Dankclaw. It replaces the Qwen3.6-27B service on
port 7770 under the same API alias. port 7770 under the same API alias.
@@ -83,18 +85,13 @@ B) switch back to Qwen3.6-27B (frank-default-large, MTP n_max 3).
Passwordless management is installed: Passwordless management is installed:
`/usr/local/sbin/frank-restart-ling.sh` and `/etc/sudoers.d/frank-ling`. `/usr/local/sbin/frank-restart-ling.sh` and `/etc/sudoers.d/frank-ling`.
## Switch Back to Qwen3.6-27B (MTP) ## Switch Back (obsolete)
The Qwen config was not changed. It remains committed as The switch-back commands below are obsolete. `frank-default-large` now
`frank-default-large.service` / `serve-llama-frank-large.sh` / runs the Qwen3.8-27B pipeline-parallel head. To restore a Ling trial,
`frank-default-large.env` (draft-mtp, n_max 3, ubatch 8192). re-enable `frank-large-ling` from the repo unit
(`scripts/frank-large-ling.service`) and serve script
Run these commands: (`scripts/serve-ling-flash.sh`), then start it on a non-conflicting port.
```
sudo systemctl stop frank-large-ling
sudo systemctl start frank-default-large
```
## Caveats ## Caveats
docs/ling-3.0-flash-vllm-plan.md
+8 -38
View File
@@ -1,40 +1,10 @@
# Ling-3.0-flash on Dankclaw — vLLM-ROCm Deployment Plan # Ling-3.0-flash on Dankclaw — vLLM-ROCm Deployment Plan (never shipped)
## Decisions (confirmed) This page was the plan for a vLLM-ROCm deployment of
`jcbtc/Ling-3.0-Flash-CIRU-int4-Strix-native` (INT4 W4A16, ~62 GB) on
Dankclaw. The deployment never shipped. The actual Ling work used the
llama.cpp Q6_K fork trial instead: [[ling-3.0-flash-trial]].
| Decision | Choice | The trial service (`frank-large-ling`) is retired and disabled under the
|----------|--------| boot policy (2026-08-17). `frank-default-large` is the Qwen3.8-27B PP
| Objective | Quality-first; correctness (tool calling) over raw quant speed | head, not the Ling path. See git history for the full decision table.
| Runtime | vLLM-ROCm on host ROCm venv (jcbtc validated gfx1151 path) |
| Model | jcbtc/Ling-3.0-Flash-CIRU-int4-Strix-native (INT4 W4A16, ~62 GB) |
| Accuracy | INT4 primary; MXFP4 (olka-fi) as controlled A/B after stability |
| KV cache | Full precision (no FP8 KV) — quality |
| Speculative decoding | MTP (native, lossless) |
| Context | Native 256k; user cap ~200k; 4-6 concurrent sessions (~8 max_num_seqs) |
| Throughput | Maximum (chunked prefill, large max_num_batched_tokens) |
| systemd unit | frank-default-large.service |
| Served alias | frank-default-large |
| Port | 7770 (8080s overused) |
| Migration | Kill llama.cpp, disable llama-server.service, reclaim memory |
| Delivery | Follow AGENTS.md git workflow; scripts/ + docs/ in repo |
## Why not alternatives
- FP8: broken/lossy on Strix Halo; no ROCmFP8 Ling model; 124 GB does not fit.
- HIGHER bpw (Q5/Q6/Q8/BF16/FP8): exceed 96 GB usable wall for a 124B MoE.
- ROCmFP4 (raulvidis): defective tool calling; llama.cpp-only concurrency limits.
- Vulkan: llama.cpp backend only, not vLLM; no quality ceiling unlock.
- FP8 KV cache: rejected for quality (full-precision KV affordable because
Ling is hybrid linear-attention with small KV footprint).
## Steps
1. Pre-flight: disk space, ROCm/amdgpu state, memory baseline.
2. Bring jcbtc scripts + frank-default-large.service into repo; commit.
3. Build ROCm vLLM (install-host-deps.sh + build-vllm-gfx1151.sh).
4. Download weights via download-model.sh (~62 GB).
5. Validate on staging: MTP engages, tool calling works, quality + throughput.
6. Cut over: kill llama.cpp, disable llama-server, enable frank-default-large on 7770.
7. Tune max throughput; confirm concurrent-token capacity.
8. MXFP4 A/B (post-stable).
9. Update AGENTS.md, README, and KB.
docs/qwen3.8-spec-matrix.md
+43 -117
View File
@@ -1,9 +1,8 @@
# Qwen3.8-27B UD-Q8_K_XL Speculative-Decoding Matrix # Qwen3.8-27B Speculative-Decoding Matrix
This page records the speculative-decoding benchmark and the production This page records the speculative-decoding benchmark history and the
tuning for the unsloth Qwen3.8-27B `UD-Q8_K_XL` service production tuning for the Qwen3.8-27B service (`frank-default-large`),
(`frank-default-large`), now a pipeline-parallel head with a Dankserver now a pipeline-parallel head with a Dankserver worker.
worker.
## Production state (2026-08-17: Q6_K, q8_0/f16 KV, split 0.68) ## Production state (2026-08-17: Q6_K, q8_0/f16 KV, split 0.68)
@@ -42,99 +41,44 @@ Current production, verified against the live process and journal
- Batch 8192 / ubatch 256, `--direct-io`, `--fit off`, - Batch 8192 / ubatch 256, `--direct-io`, `--fit off`,
`--gpu-layers all`. `--gpu-layers all`.
## Result (superseded 2026-08-17: Q8_K_XL, f16/f16 KV, split 0.53) ## How MTP works on Qwen3.8 (bundled-head era)
Superseded for topology on 2026-08-16, when the PP split became
production. The ngram-mod preset survives as the fallback layer under
MTP. The Vulkan0 verdict in this section applies to the standalone
configuration only.
- Model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB, bundled MTP).
- Spec: llama.cpp **`--spec-default`** preset = **ngram-mod** (n_match 24,
n_min 48, n_max 64). No draft model. Chosen 2026-08-15 after DSpark
crashed under parallel load (see below).
- **PARALLEL=2** — passes the 2x32k parallel stream test (both streams
complete 256/256 concurrently, no crash). ngram-mod is draftless and
parallel-safe.
- **Backend: Vulkan0** — at 262144 ctx (~200k prompt) Vulkan decodes
5.68 t/s vs ROCm 3.83 t/s (48% faster); ROCm only wins prefill (119 vs
106 t/s). See the high-context bench table below.
- Batch 8192 / ubatch 256, ctx 524288 (262144 per slot).
## Result (superseded 2026-08-15: DSpark)
- Model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB, bundled MTP).
- Spec: **DSpark sidecar draft** `magnitudedev/Qwen3.8-27B-DSpark-GGUF`
(`Qwen3.8-27B-DSpark-Q8_0.gguf`, 1.36B, block_size 7), `--spec-type
draft-dspark`, `--spec-draft-n-max 7` (clamps to block_size).
- **PARALLEL=1.** Two concurrent 16k prose prompts crashed the ROCm backend
with `ROCm error: an illegal memory access` during per-sequence block
sampling. See the DSpark parallel incident below.
- Retired the same day in favor of `--spec-default`.
- Batch 8192 / ubatch 256, ctx 524288 (262144 per slot).
## Result (superseded 2026-08-14: bundled MTP)
- Model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB, bundled MTP).
- Spec: `draft-mtp`, `--spec-draft-n-max 5`, no `--spec-draft-p-min`.
- Batch 8192 / ubatch 256, parallel 2, ctx 524288 (262144 per slot).
- Realistic-prompt generation: ~13.3 t/s single, ~8 t/s per slot under
2 concurrent streams. No premature stop.
## Incident: DSpark ROCm Crash Under Parallel Load (2026-08-15)
A two-stream parallel acceptance test crashed `frank-default-large`.
```mermaid
graph TD
A[Two concurrent 16k prose prompts<br>high prompt-cache reuse] --> B[ROCm illegal memory access<br>in common_speculative_impl_draft_dflash::draft]
B --> C[status 6/ABRT core dump]
C --> D[systemd auto-restart, 503 during reload]
D --> E[pinned PARALLEL=1]
```
Trigger: two distinct-but-overlapping ~16k-token realistic prose prompts
started simultaneously on PARALLEL=2. The ROCm backend aborted at
`ggml-cuda.cu:106` with `ROCm error: an illegal memory access was
encountered`, surfacing in `hipStreamSynchronize` inside
`common_speculative_impl_draft_dflash::draft` (the class used by
`draft-dspark`). `systemd` caught it (status 6/ABRT) and restarted the
unit; in-flight streams saw `Remote end closed connection without response`
until the ~reload finished.
Mechanism: DSpark reuses the DFlash block-diffusion draft machinery. The
DFlash draft model uses one shared draft context (`ctx_dft`) across all
parallel slots; under concurrent block drafting on ROCm it performs an
illegal memory access (HIP stream sync). This is the same multi-slot
class of bug that retired `draft-dflash` for premature EOS; here it is a
hard ROCm crash instead of a bad sample.
Fix: pin **PARALLEL=1** (single slot). With one slot, concurrent requests
queue safely and each completes to `max_tokens`. Revisit after an upstream
multi-slot DSpark fix (DSpark is known-good single-slot). Single-slot 16k
prose generation was ~steady (measuring only the correct-size signal, not
this page).
## Parallel results by stream count (2026-08-15, correct-size tests)
| Parallel | Two 16k prose prompts | Two 67k prose prompts |
| -------- | -------------------- | --------------------- |
| 2 | crashes (illegal mem access) | passes |
| 1 | queues, both complete | n/a |
Note: the crash manifests on the smaller (16k) concurrent streams, not the
larger 67k pair. Size is not the trigger; concurrent block drafting on the
shared draft context is.
## How MTP works on Qwen3.8
The unsloth `UD-*` GGUFs keep the MTP (Multi-Token Prediction) head inside The unsloth `UD-*` GGUFs keep the MTP (Multi-Token Prediction) head inside
the model file. llama.cpp b10411+ auto-detects it and creates the draft the model file. llama.cpp b10411+ auto-detects it and creates the draft
context from the target model. Do not pass `--spec-draft-model`. context from the target model. Do not pass `--spec-draft-model` for those
files. This is superseded: the production `Qwen3.8-27B-Uncensored-Q6_K.gguf`
has NO bundled head, and the serve script passes the standalone draft
sidecar with `--spec-draft-model` (see Production state above).
The stock llama.cpp build must be b10411 or newer. The 2026-08-14 rebuild The stock llama.cpp build must be b10411 or newer. The 2026-08-14 rebuild
on Dankclaw is b10433 (`9b05354ec`) with `GGML_HIP` + `gfx1151` + Vulkan. on Dankclaw is b10433 (`9b05354ec`) with `GGML_HIP` + `gfx1151` + Vulkan.
## History (superseded configurations, 2026-08-14 to 08-16)
All superseded configs used `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB,
bundled MTP) before the Q6_K swap. The ngram-mod preset survives as the
fallback layer under MTP. The Vulkan0 verdict from this era applies to the
standalone configuration only; the PP head runs ROCm0.
- **2026-08-14: bundled MTP** — `draft-mtp`, n_max 5, parallel 2, ctx
524288 (2 x 262144). ~13.3 t/s single, ~8 t/s per slot under 2
concurrent streams. No premature stop.
- **2026-08-15: DSpark sidecar** — `magnitudedev/Qwen3.8-27B-DSpark-GGUF`
(Q8_0, 1.36B, block_size 7), n_max 7, PARALLEL=1. Two concurrent 16k
prose prompts crashed the ROCm backend with `ROCm error: an illegal
memory access` in `common_speculative_impl_draft_dflash::draft` (status
6/ABRT, systemd restart, ~503 during reload). DSpark reuses the DFlash
block-diffusion machinery, which shares one draft context across
parallel slots; concurrent block drafting faults in HIP stream sync.
This is the same multi-slot bug class that retired `draft-dflash`
(premature EOS). The crash shows on 16k concurrent streams, not on a
67k pair — concurrency is the trigger, not size. Fix: PARALLEL=1
(requests queue safely). Retired the same day.
- **2026-08-15 to 08-16: `--spec-default` (ngram-mod), Vulkan0** — n_match
24 / n_min 48 / n_max 64, draftless, PARALLEL=2 (2 x 32k parallel stream
test passes). Chosen for parallel safety after DSpark; kept as the
fallback under MTP in the current config.
## Benchmark Method ## Benchmark Method
The matrix ran 54 configurations: 3 spec types (`draft-mtp,ngram-mod`, The matrix ran 54 configurations: 3 spec types (`draft-mtp,ngram-mod`,
@@ -189,37 +133,19 @@ ubatch 256, single slot, on scratch port 7892.
Verdict: at very high context, Vulkan0 decodes **48% faster** (5.68 vs Verdict: at very high context, Vulkan0 decodes **48% faster** (5.68 vs
3.83 t/s); ROCm0 prefills 12% faster. Generation is the interactive cost 3.83 t/s); ROCm0 prefills 12% faster. Generation is the interactive cost
(users stream tokens), so **Vulkan0** is the production backend. Both (users stream tokens). This verdict applies to the standalone
`tg_3s` values were rock-steady across the run (3.83 and 5.67 t/s). configuration; the PP head runs ROCm0 (HIP registers before Vulkan).
Both `tg_3s` values were rock-steady across the run (3.83 and 5.67 t/s).
High-context decode collapses on both backends versus ~25 t/s at 16k ctx; High-context decode collapses on both backends versus ~25 t/s at 16k ctx;
that is the long-context attention penalty, not a spec artifact. that is the long-context attention penalty, not a spec artifact.
## Production Configuration ## Production Configuration
`scripts/frank-default-large.env` (live 2026-08-16): The live tunables live in `scripts/frank-default-large.env` (head) and
`scripts/frank-pp-worker.env` (worker) in this repo. The Production
```ini state section at the top of this page records the current values: Q6_K
RPC_SERVERS=192.168.0.235:50052 # Dankserver worker model + standalone MTP draft, split 0.68/0.32, 262144 ctx, PARALLEL=1,
TENSOR_SPLIT=0.40,0.60 # [RPC0, iGPU] q8_0/f16 KV.
CTX_SIZE=401400 # pool = 2 slots x 200700
PARALLEL=2
SPEC_NONE=0
SPEC_DEFAULT=1 # ngram-mod fallback (48/64/24)
MTP=1 # draft-mtp, bundled head
SPEC_DRAFT_N_MAX=4
SPEC_NGRAM_MOD_N_MIN=48
SPEC_NGRAM_MOD_N_MAX=64
SPEC_NGRAM_MOD_N_MATCH=24
CACHE_TYPE_K=f16
CACHE_TYPE_V=f16
BATCH_SIZE=8192
UBATCH_SIZE=256
```
There is no `DEVICE` variable. The head must not pass `--device` (the RPC
device would be silently dropped). Live slot context: `n_ctx_slot =
200704`. The pi `models.json` contextWindow for `frank-default-large`
declares 200000, a safe under-estimate of the live slot.
## See Also ## See Also