docs: boot policy (only PP units enabled), deploy.sh dankserver hazard, stale-claim cleanup
- deployment.md: Boot Policy (2026-08-17) section (disabled units table, re-enable recipes, stale frank-restart-small.sh helper, broken deploy.sh dankserver target); Update Dankserver -> frank-pp-worker; condensed retired 35B history block and 2026-08-13 incident - AGENTS.md: boot-policy note, target map fix (Q6_K, split 0.68), deploy.sh dankserver warning, obsolete quirk entries condensed - README.md: rewrite active services/models tables (PP head + worker) - index.md: topology fix (qwythos:9b, Q6_K split 0.68, NPU disabled) - qwen3.8-spec-matrix.md: superseded configs merged into compact history; stale inline env block removed; bundled-head MTP note - FastFlowLM, PP Deployment: boot-policy + enablement notes - System Specs x2, DFlash doc, 35B FP8 doc, Ling docs: stale active claims fixed, retired banners, never-shipped plan stubbed
This commit is contained in:
Frank
@@ -15,11 +15,11 @@ The Dankclaw NPU endpoint (FastFlowLM, `frank-default-small`, port 52625)
|
||||
serves Qwythos-9B-Claude-Mythos-5-1M (abliterated) as registry tag
|
||||
`qwythos:9b` at 262144 ctx (swapped in 2026-08-17; previously
|
||||
`qwen3.6-moe:35b-a3b`; the old llama.cpp 35B service on Dankserver is
|
||||
stopped). See [docs/deployment.md](docs/deployment.md) for the deployment
|
||||
process, [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md) for the
|
||||
spec history and current production state, and
|
||||
[docs/Pipeline Parallelism/Qwen3.8 PP Deployment.md](docs/Pipeline%20Parallelism/Qwen3.8%20PP%20Deployment.md)
|
||||
for the PP runbook.
|
||||
stopped). **Boot policy (2026-08-17):** only `frank-default-large` and
|
||||
`frank-pp-worker` are enabled for boot. The NPU unit `frank-default-small`
|
||||
is disabled; the running service keeps serving until stopped or rebooted.
|
||||
See [docs/deployment.md](docs/deployment.md) for the deployment
|
||||
process, the boot policy, and the re-enable recipes.
|
||||
|
||||
**Sibling project:** [../serverAppConfigs](../serverAppConfigs) manages the
|
||||
Podman app stack on the same hosts (OpenWebUI, Gitea, observability). Its
|
||||
@@ -83,10 +83,14 @@ Target map:
|
||||
|
||||
| Target | Host | Runtime | Service | Port |
|
||||
|--------|------|---------|---------|------|
|
||||
| Dankclaw | dank@dankclaw | llama.cpp PP head (Qwen3.8-27B UD-Q8_K_XL, split 0.40/0.60 to the Dankserver worker via RPC, MTP+ngram-mod, f16 KV) | frank-default-large | 7770 |
|
||||
| Dankclaw | dank@dankclaw | FastFlowLM NPU (XDNA2 50 TOPS, qwythos:9b Q4NX 7.76 GB, 262144 ctx; see [[FastFlowLM NPU Inference]]) | frank-default-small | 52625 |
|
||||
| Dankclaw | dank@dankclaw | llama.cpp PP head (Qwen3.8-27B-Uncensored Q6_K, split 0.68/0.32 to the Dankserver worker via RPC, standalone MTP draft + ngram-mod, q8_0/f16 KV) | frank-default-large | 7770 |
|
||||
| Dankclaw | dank@dankclaw | FastFlowLM NPU (XDNA2 50 TOPS, qwythos:9b Q4NX 7.76 GB, 262144 ctx; see [[FastFlowLM NPU Inference]]) — disabled for boot since 2026-08-17 | frank-default-small | 52625 |
|
||||
| Dankserver | dank@dankserver | ggml-rpc worker (RTX 3090, receives the head's tensor share) | frank-pp-worker | 50052 |
|
||||
|
||||
> Warning: `deploy.sh dankserver` is currently broken. It targets the
|
||||
> removed `frank-default-small` unit. Restart the worker with
|
||||
> `sudo systemctl restart frank-pp-worker` until the script is fixed.
|
||||
|
||||
## Runtime & Troubleshooting
|
||||
|
||||
### Env var naming rule — HIP runtime collision
|
||||
@@ -117,7 +121,7 @@ graph TD
|
||||
2. Run the serve script directly. Then run the raw binary with the same flags. A different outcome isolates the env file sourcing.
|
||||
3. Run gdb for the backtrace with this command:
|
||||
```
|
||||
gdb -batch -ex run -ex "bt 25" --args bash scripts/serve-llama-frank-large.sh
|
||||
gdb -batch -ex run -ex "bt 25" --args bash scripts/serve-pp-head.sh
|
||||
```
|
||||
`llama-server` buffers its timed stdout. On SIGSEGV the log tail is lost. Trust gdb, not the log.
|
||||
4. Kernel spam about `amdgpu: SVM mapping failed` is a symptom, not the root cause. Investigate other causes before assuming memory pressure.
|
||||
@@ -125,13 +129,13 @@ graph TD
|
||||
### Operational quirks
|
||||
|
||||
- `sudo` on both boxes needs a tty. The remote sudo steps in `deploy.sh` fail non-interactively. Use `echo '<password>' | sudo -S <command>`.
|
||||
- Passwordless restarts: each target has a root-owned helper (`/usr/local/sbin/frank-restart-large.sh` on Dankclaw, `frank-restart-small.sh` on Dankserver) and a sudoers NOPASSWD drop-in (`/etc/sudoers.d/frank-*`). `deploy.sh` prefers the helper. The sudoers rules match commands exactly: no extra arguments (for example, `--no-pager` breaks the `systemctl status` rule). See [docs/deployment.md](docs/deployment.md).
|
||||
- Passwordless restarts: each target has a root-owned helper and a sudoers NOPASSWD drop-in (`/etc/sudoers.d/frank-*`). The Dankclaw helper `/usr/local/sbin/frank-restart-large.sh` is correct for the PP head. The Dankserver helper `frank-restart-small.sh` is **obsolete** — it restarts the removed `frank-default-small` unit. Restart the worker with `sudo systemctl restart frank-pp-worker`. The sudoers rules match commands exactly: no extra arguments (for example, `--no-pager` breaks the `systemctl status` rule). See [docs/deployment.md](docs/deployment.md).
|
||||
- Never run `pkill -f` with a pattern that appears in your own command line. It kills your ssh session. Use `pkill -x <exact-name>`.
|
||||
- Model load is slow: about 3 minutes warm, 8-17 minutes cold (measured 17 minutes on 2026-08-13 after a cache drop). Health returns `503 Loading model` the whole time. That is normal. Do not kill the process. `deploy.sh` waits up to 20 minutes on Dankclaw (see `HEALTH_WAIT`).
|
||||
- Page-cache state confounds crash tests. Cold-cache loads are slow and survive. Warm-cache loads crash fast. Control for it when testing. `deploy.sh` drops caches on Dankclaw before restart (see `DROP_CACHES`); manual restarts must run `sync && echo 3 | sudo tee /proc/sys/vm/drop_caches` first.
|
||||
- A client abort and a server release look different in the journal. A client abort logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in [docs/deployment.md](docs/deployment.md).
|
||||
- Dankserver crash signature under parallel load: `CUDA error: an unsupported value or parameter was passed to the function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541`, then `status=6/ABRT`, then `503 Loading model` for ~5 minutes. Happened twice on 2026-08-13 under a 5-researcher subagent storm (llama.cpp b10121, driver 595.84). Root cause not yet determined; do not confuse it with speculative decoding.
|
||||
- DFlash premature EOS under parallel load: on Dankclaw, `draft-dflash` ends a generating slot ~1s after a new task lands with high prompt-cache reuse (LCP ~0.99), with `finish_reason "stop"` and no `cancel task` log. It is a DFlash shared-draft-context bug. DFlash is retired from Dankclaw. DSpark (`draft-dspark`, its semi-autoregressive successor) is also unsafe here: two concurrent 16k prose prompts crashed the ROCm backend with `ROCm error: an illegal memory access` (status 6/ABRT). Production uses `--spec-default` (ngram-mod, draftless) at PARALLEL=2 on the Vulkan0 backend. See [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md).
|
||||
- Dankserver crash signature (historical): the retired 35B service aborted under parallel load with `CUDA error: an unsupported value or parameter was passed to the function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541`, then `status=6/ABRT`, then `503 Loading model` for ~5 minutes (llama.cpp b10121, driver 595.84). Root cause was never determined; the service is retired. Do not confuse that crash with speculative decoding. See [docs/dankserver-vram-growth.md](docs/dankserver-vram-growth.md).
|
||||
- DFlash premature EOS under parallel load (historical): on Dankclaw, `draft-dflash` ended a generating slot ~1s after a new task landed with high prompt-cache reuse (LCP ~0.99), with `finish_reason "stop"` and no `cancel task` log — a DFlash shared-draft-context bug. DFlash and DSpark are retired from Dankclaw. Production uses the standalone MTP draft over the ngram-mod fallback at PARALLEL=1 on ROCm0. See [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md).
|
||||
|
||||
### Dankclaw memory model
|
||||
|
||||
|
||||
@@ -1,11 +1,13 @@
|
||||
# vLLM Scripts
|
||||
|
||||
Scripts for running LLM inference on two local machines.
|
||||
Deployment scripts for the LLM inference fleet: a llama.cpp
|
||||
pipeline-parallel Qwen3.8-27B split across two machines, plus the
|
||||
FastFlowLM NPU endpoint on Dankclaw.
|
||||
|
||||
## Targets
|
||||
|
||||
| Host | User | Codename | Architecture |
|
||||
|------|------|----------|-------------|
|
||||
|------|------|----------|--------------|
|
||||
| dankserver | dank | Dankserver | NVIDIA RTX 3090 + 62 GB RAM |
|
||||
| dankclaw | dank | Dankclaw | AMD Strix Halo APU (shared memory) |
|
||||
|
||||
@@ -13,30 +15,32 @@ Scripts for running LLM inference on two local machines.
|
||||
|
||||
| Service | Host | Runtime | Model | Port |
|
||||
|---------|------|---------|-------|------|
|
||||
| frank-default-large | dankclaw | vLLM-ROCm (CIRU) | Ling-3.0-flash CIRU INT4 MTP | 7770 |
|
||||
| frank-default-small | dankserver | llama.cpp (ngram-mod) | Qwen3.6-35B-A3B uncensored Q6_K_P | 7770 |
|
||||
| frank-default-large | dankclaw | llama.cpp PP head (RPC to Dankserver) | Qwen3.8-27B-Uncensored Q6_K, standalone MTP draft | 7770 |
|
||||
| frank-pp-worker | dankserver | ggml-rpc worker (RTX 3090) | receives the head's tensor share | 50052 |
|
||||
|
||||
> The retired ROCmFPX llama.cpp service on Dankclaw was replaced by the vLLM service above. The retired `vllm-awq-qwen36` (AWQ 4-bit) service on Dankserver was replaced by the llama.cpp service above. Legacy unit files remain in `scripts/` for reference.
|
||||
|
||||
## Models
|
||||
|
||||
| Model | Target | Format |
|
||||
|-------|--------|--------|
|
||||
| Ling-3.0-flash CIRU INT4 (MTP) | dankclaw | INT4 W4A16 (Quark/vLLM) |
|
||||
| Qwen3.6 35B A3B uncensored Q6_K_P (ngram-mod speculative decoding) | dankserver | GGUF Q6_K_P (llama.cpp, MoE offload) |
|
||||
| Qwen3.6 35B A3B AWQ 4bit | dankserver | AWQ (retired) |
|
||||
| Qwen3.6 35B A3B FP8 | dankserver | FP8 (documented, not active) |
|
||||
| Qwen3.6 27B FP8 | dankclaw | FP8 (documented, not active) |
|
||||
Boot policy (2026-08-17): only these two units are enabled. The
|
||||
FastFlowLM NPU endpoint (`frank-default-small`, qwythos:9b, port 52625)
|
||||
is disabled for boot; re-enable it from `scripts/frank-default-small.service`
|
||||
if needed. Retired units: the Dankserver llama.cpp 35B service,
|
||||
`llamaserver`, `ollama`, `vllm-awq-qwen36`, `vllm-qwen36`,
|
||||
`frank-large-ling` (Ling trial), and the vLLM-ROCm CIRU path. Their unit
|
||||
files remain in `scripts/` for reference.
|
||||
|
||||
## Structure
|
||||
|
||||
- `docs/` — Project knowledge base with research notes on vLLM configuration, hardware specifics, speculative decoding, and deployment notes
|
||||
- `scripts/` — Deployment and launch scripts (deploy.sh, serve-llama-frank-small.sh, serve-ling3-vllm.sh, systemd unit files)
|
||||
- `docs/` — knowledge base: deployment runbook, PP runbook, spec matrix,
|
||||
NPU notes, research archive
|
||||
- `scripts/` — serve scripts, env files, systemd units, deploy and test
|
||||
tools
|
||||
|
||||
## Deployment
|
||||
|
||||
See [docs/deployment.md](docs/deployment.md) for the deploy/update process. Run `./scripts/deploy.sh <dankclaw|dankserver>` after pushing committed changes.
|
||||
See [docs/deployment.md](docs/deployment.md). Run
|
||||
`./scripts/deploy.sh dankclaw` for the PP head. The `dankserver` target
|
||||
is broken (it targets the removed `frank-default-small` unit); restart
|
||||
the worker with `sudo systemctl restart frank-pp-worker` until the
|
||||
script is fixed.
|
||||
|
||||
## Knowledge Base
|
||||
|
||||
See [docs/index.md](docs/index.md) for the project knowledge base index.
|
||||
See [docs/index.md](docs/index.md) for the project knowledge base index.
|
||||
|
||||
@@ -160,9 +160,7 @@ The DFlash draft is not a standalone model. It must pair with its target model.
|
||||
|
||||
## Deployment on Dankserver (NVIDIA RTX 3090)
|
||||
|
||||
## Deployment on Dankserver (NVIDIA RTX 3090)
|
||||
|
||||
The active Dankserver deployment runs **llama.cpp** (`frank-default-small`), not vLLM. It serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` on port 7770 using **ngram-mod** speculative decoding (n_min=48 / n_max=64 / n_match=24) with 3 slots at 200000 context each and MoE experts on CPU. See [[deployment]] and `scripts/serve-llama-frank-small.sh`.
|
||||
The Dankserver llama.cpp deployment (`frank-default-small`, Qwen3.6-35B-A3B) is retired (2026-08-16) and its unit disabled (2026-08-17). The RTX 3090 now runs the PP worker for the Dankclaw head (see [[Qwen3.8 PP Deployment]]). The vLLM material below documents the archived AWQ/DFlash path on the same hardware.
|
||||
|
||||
> **DFlash retired from active service.** `DFlash` (draft-dflash) was tested initially but did not deliver the desired speed gains under the `frank-default-small` workload (see ngram-mod benchmarks below). The `Alittlehammmer/Qwen3.6-35B-A3B-DFlash` draft GGUF remains in `/home/dank/models/` as reference material only.
|
||||
|
||||
@@ -197,15 +195,7 @@ docker run --rm --name vllm-qwen36-dflash \
|
||||
|
||||
## Deployment on Dankclaw (AMD Strix Halo gfx1151)
|
||||
|
||||
Dankclaw now runs llama.cpp as its runtime for `frank-default-large` (port 7770), serving the unsloth Qwen3.8-27B `UD-Q8_K_XL` model with **no dedicated draft** on the **Vulkan0** backend.
|
||||
|
||||
Current facts (2026-08-15):
|
||||
|
||||
- Main model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (bundled MTP, unused).
|
||||
- Spec: llama.cpp **`--spec-default`** preset = ngram-mod (n_match 24 / n_min 48 / n_max 64). No draft model.
|
||||
- **Backend: Vulkan0** — at 262144 ctx (~200k prompt) Vulkan decodes 5.68 t/s vs ROCm 3.83 t/s (48% faster); ROCm wins prefill 119 vs 106 t/s. See [[qwen3.8-spec-matrix]] for the bench table.
|
||||
- **PARALLEL=2** — passes the 2x32k parallel stream test.
|
||||
- KV `q8_0` K / `q8_0` V; context 2 x 262144; batch 8192 / ubatch 256.
|
||||
`frank-default-large` on Dankclaw runs the llama.cpp pipeline-parallel head (Qwen3.8-27B-Uncensored Q6_K) on port 7770, RPC-split with the Dankserver worker. Spec is the standalone MTP draft over the ngram-mod fallback, 1 slot at 262144 ctx, q8_0/f16 KV, head GPU side ROCm0. See [[qwen3.8-spec-matrix]] and [[Qwen3.8 PP Deployment]] for the current state and the 2026-08-15 Vulkan0/ROCm0 high-context bench.
|
||||
|
||||
History: before `--spec-default`, the box ran DSpark (`draft-dspark`, retired 2026-08-15 after a ROCm illegal-memory-access crash under two concurrent 16k prompts) and before that bundled MTP (`draft-mtp`, retired same day). DFlash was retired on 2026-08-14 (premature-EOS under parallel load). The old DFlash draft GGUF (`Qwen3.6-27B-DFlash-Q8_0.gguf`), the MTP-only drafter (`Qwen3.6-27B-MTP-ONLY-Q8_0.gguf`), and the DSpark draft (`Qwen3.8-27B-DSpark-Q8_0.gguf`) remain in `/home/dank/models/` as reference only.
|
||||
|
||||
|
||||
@@ -82,7 +82,7 @@ The NPU shares the 30 GiB unified pool. FLM allocates roughly half of system RAM
|
||||
- ROCm reports 96 GiB of GPU-visible memory (overcommitted). The current vLLM service runs at `--gpu-memory-utilization 0.95`, which fits the CIRU INT4 weights and 256k KV cache (about 90 GiB used of the 96 GiB frame buffer).
|
||||
- Without the ROCm VRAM patch, ROCm clamps APU memory to 50% (see `scripts/patches/rocm_patched.py` on the remote). The patch restores the real GTT ceiling minus an 8 GB safety margin.
|
||||
- Only 8 GiB swap. Modern vLLM uses `--kv-offloading-size` instead of the removed `--swap-space` flag.
|
||||
- The active service on Dankclaw is `frank-default-large.service`: vLLM-ROCm serving the Ling-3.0-flash CIRU INT4 (W4A16) model on port 7770 with native MTP speculative decoding. It replaced the ROCmFPX llama.cpp service.
|
||||
- The active service on Dankclaw is `frank-default-large.service`: the llama.cpp pipeline-parallel head (Qwen3.8-27B-Uncensored Q6_K) on port 7770, RPC-split with the Dankserver worker. The vLLM-ROCm CIRU Ling path was planned but never shipped. See [[deployment]].
|
||||
- ROCm 7.2.4 from AMD repos. The CIRU venv bundles a self-contained ROCm 7.15 toolchain; vLLM ROCm support requires ROCm 6.3+.
|
||||
- Python 3.14.4 is very new. vLLM ROCm wheels target Python 3.12. The CIRU venv uses Python 3.12.
|
||||
- Radeon 8060S falls under Ryzen AI MAX series (gfx1151). vLLM lists this as supported.
|
||||
|
||||
@@ -51,11 +51,11 @@ Dankserver is an NVIDIA GPU server for vLLM inference. The host runs Ubuntu 24.0
|
||||
|
||||
### Active Services
|
||||
|
||||
The active `frank-default-small` service (llama.cpp, `b10108+`) serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model on port 7770 with **ngram-mod** speculative decoding. It runs **4 parallel slots** at 175104 context each (pool 700416, refitted 2026-08-14 for the dual-f16 KV cache). Of the 40 layers' MoE expert weights, **36 sit on CPU** (`--n-cpu-moe 36`) while the remaining **4 layers' expert weights** are resident on GPU along with all non-expert tensors (`--gpu-layers all`). KV stays on GPU with K=`f16`, V=`f16` (13680 MiB pool at 700416 cells). Batch 8192 / ubatch 3072, `--ctx-checkpoints 4`, `--no-mmap`. At idle ~22.1 GB VRAM used, leaving ~2.0 GiB headroom. Idle RAM ~32 GB of 62 GB.
|
||||
The active service on Dankserver is `frank-pp-worker.service`: the ggml-rpc worker (RTX 3090) that receives the PP head's tensor share (45 of 65 blocks, ~23.8 GiB of 24 GiB at steady state). The old llama.cpp Qwen3.6-35B-A3B service (`frank-default-small`) is retired (2026-08-16) and disabled (2026-08-17). See [[Qwen3.8 PP Deployment]].
|
||||
|
||||
## Hardware Relevance
|
||||
|
||||
- 24 GB VRAM is close to saturated with this config. At idle ~22.1 GB used, leaving ~2.0 GiB headroom at `--n-cpu-moe 36` (4 layers' experts on GPU) with `ubatch 3072` and a dual-f16 KV pool of 13680 MiB at 700416 cells (2026-08-14 refit; the same pool at 800768 cells OOM'd the load warmup). A llama-bench sweep ([[frank-default-small-benchmark]]) guided this state: prefill is the priority, so `ubatch 3072` stays (top prefill tier), and `ncmoe 36` was raised from 35 to fix a cuBLAS OOM-class crash during big prefills by freeing ~0.75 GiB peak VRAM at only -2% prefill. If it still crashes under load, add `GGML_CUDA_FORCE_MMQ=1`. If decode speed matters more, use `ncmoe 32 - ubatch 1024` (gen ~48.7 t/s). By editing `scripts/frank-default-small.env`.
|
||||
- 24 GB VRAM is close to saturated with this config. At idle ~22.1 GB used, leaving ~2.0 GiB headroom at `--n-cpu-moe 36` (4 layers' experts on GPU) with `ubatch 3072` and a dual-f16 KV pool of 13680 MiB at 700416 cells (2026-08-14 refit; the same pool at 800768 cells OOM'd the load warmup). A llama-bench sweep ([[frank-default-small-benchmark]]) guided this state: prefill is the priority, so `ubatch 3072` stays (top prefill tier), and `ncmoe 36` was raised from 35 to fix a cuBLAS OOM-class crash during big prefills by freeing ~0.75 GiB peak VRAM at only -2% prefill. If it still crashes under load, add `GGML_CUDA_FORCE_MMQ=1`. If decode speed matters more, use `ncmoe 32 - ubatch 1024` (gen ~48.7 t/s). The tuning guidance applied to the retired `frank-default-small` service; the PP worker's allocations are fixed at load.
|
||||
- 62 GB system RAM holds the ~29 GB of MoE expert weights with room to spare (~57 GB available after load).
|
||||
- 127 GB swap is available for CPU KV cache paging if GPU KV offload is disabled.
|
||||
- Root disk has approximately 140 GB free. Store the GGUF models under `/home/dank/models` (about 32 GB total).
|
||||
|
||||
@@ -200,6 +200,16 @@ Notes:
|
||||
|
||||
`frank-default-small` now runs the NPU endpoint on Dankclaw (was the Dankserver llama.cpp service). Unit: `scripts/frank-default-small.service` (symlinked into `/etc/systemd/system`), wrapper `scripts/serve-flm-small.sh`, tunables `scripts/frank-default-small.env`. `LimitMEMLOCK=infinity` is set in the unit. Default model: `gemma4-it:e4b`. Swap `FLM_MODEL` in the env to switch.
|
||||
|
||||
### Boot policy (2026-08-17)
|
||||
|
||||
The `frank-default-small` unit is disabled for boot. Only the PP pair (`frank-default-large` + `frank-pp-worker`) starts at boot. The unit file was removed from `/etc/systemd/system`; the running FLM service keeps serving until stopped or rebooted. To restore the endpoint, run these commands on Dankclaw:
|
||||
|
||||
```
|
||||
sudo ln -sf /home/dank/vllmScripts/scripts/frank-default-small.service /etc/systemd/system/frank-default-small.service
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable frank-default-small
|
||||
```
|
||||
|
||||
Live config (2026-08-16): `FLM_PMODE=turbo`, `FLM_ASR=1` (loads Whisper-V3-Turbo-NPU2; exposes `/v1/audio/transcriptions`), `FLM_CTX_LEN=131072` (verified max — Gemma4 natively supports 128K and the NPU budget fits it; KV occupancy measured at 0.11% for a 123-token exchange, decode 12.0 t/s, 13 GiB headroom).
|
||||
|
||||
The PP split was renamed `frank-pp-head` → `frank-default-large` (2026-08-16, commit `48142f5`).
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Qwen3.8-27B Pipeline-Parallel Deployment (Dankclaw + Dankserver)
|
||||
|
||||
Split Qwen3.8-27B UD-Q8_K_XL across the Strix Halo iGPU (head) and the RTX
|
||||
Split Qwen3.8-27B across the Strix Halo iGPU (head) and the RTX
|
||||
3090 (worker) using llama.cpp RPC. This page is the design record and the
|
||||
morning-of runbook. Research background:
|
||||
[Qwen3.8 Strix optimizations](../../qwen3.8-strix-optimizations.md).
|
||||
@@ -31,6 +31,19 @@ state, verified against the live journal and output smoke tests
|
||||
- Unit `frank-default-large.service`, launched through
|
||||
`serve-pp-head.sh` + `frank-default-large.env`.
|
||||
|
||||
## Service enablement (2026-08-17)
|
||||
|
||||
The PP pair is the only inference service enabled for boot. Both units
|
||||
(`frank-default-large` on Dankclaw, `frank-pp-worker` on Dankserver) are
|
||||
enabled. The NPU unit (`frank-default-small` on Dankclaw) and all retired
|
||||
units are disabled. Restart order: worker first, then head. The Dankserver
|
||||
helper `frank-restart-small.sh` is obsolete (it restarts the removed
|
||||
`frank-default-small` unit); restart the worker with
|
||||
`sudo systemctl restart frank-pp-worker`. The Dankclaw helper
|
||||
`frank-restart-large.sh` (sync + cache drop + restart) is correct for the
|
||||
head. `deploy.sh dankserver` is broken until retargeted. See
|
||||
[[deployment]] for the full boot-policy table.
|
||||
|
||||
## FA-all-quants rebuild (2026-08-17)
|
||||
|
||||
Flash Attention with mismatched K/V cache types requires the
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
# Running Qwen3.6 35B A3B FP8 on Dankserver
|
||||
|
||||
> **RETIRED.** This page documents the FP8 vLLM path for Qwen3.6-35B-A3B on Dankserver. The llama.cpp 35B service is retired (2026-08-16), its unit file was removed (2026-08-17), and the FP8 vLLM unit (`vllm-qwen36`) never shipped and is disabled. The content is kept as research history. The RTX 3090 now runs the PP worker: see [[Qwen3.8 PP Deployment]].
|
||||
|
||||
## Model Information
|
||||
|
||||
Qwen3.6-35B-A3B is a Mixture-of-Experts vision-language model from the Qwen3.6 series. It uses 35B total parameters with only 3B activated per token.
|
||||
@@ -112,10 +114,9 @@ export HF_HOME=/mnt/entropy/huggingface
|
||||
|
||||
### Active Service Conflict
|
||||
|
||||
The active `frank-default-small` service (llama.cpp, port 7770) uses about 22 GB of the 24 GB VRAM. The retired `vllm-awq-qwen36` (Podman, ~22 GB) and `vllm-qwen36` services are not active. Stop any competing service before starting another:
|
||||
The retired `frank-default-small` service (llama.cpp) used about 22 GB of the 24 GB VRAM. The current PP worker holds ~23.8 GiB. The `vllm-awq-qwen36` (Podman) and `vllm-qwen36` units are disabled. Before starting any research service, verify that the GPU is free:
|
||||
|
||||
```
|
||||
podman stop vllm-qwen36-dflash
|
||||
nvidia-smi # verify GPU is free
|
||||
```
|
||||
|
||||
|
||||
+53
-26
@@ -8,11 +8,43 @@ Two servers run LLM inference services. Each server pulls its scripts from this
|
||||
|
||||
| Target | Host | Runtime | Service | Port |
|
||||
|--------|------|---------|---------|------|
|
||||
| Dankclaw | dank@dankclaw | llama.cpp PP head (Qwen3.8-27B UD-Q8_K_XL) | frank-default-large | 7770 |
|
||||
| Dankclaw | dank@dankclaw | llama.cpp PP head (Qwen3.8-27B-Uncensored Q6_K, split 0.68 to the Dankserver worker) | frank-default-large | 7770 |
|
||||
| Dankclaw | dank@dankclaw | FastFlowLM NPU (qwythos:9b, 262144 ctx) | frank-default-small | 52625 |
|
||||
| Dankserver | dank@dankserver | ggml-rpc worker (RTX 3090) | frank-pp-worker | 50052 |
|
||||
|
||||
The `frank-default-small` (NPU) unit is not in `deploy.sh`. Restart it by hand with `sudo systemctl restart frank-default-small`.
|
||||
The `frank-default-small` (NPU) unit is not in `deploy.sh` and is disabled for boot (see Boot Policy below). If the unit is still loaded (pre-reboot), restart it by hand with `sudo systemctl restart frank-default-small`. After a reboot, re-enable the unit first (see Boot Policy).
|
||||
|
||||
## Boot Policy (2026-08-17)
|
||||
|
||||
Only the pipeline-parallel pair starts at boot: `frank-default-large` (Dankclaw head) and `frank-pp-worker` (Dankserver worker). Every other inference unit is disabled. This was set on 2026-08-17.
|
||||
|
||||
Disabled units:
|
||||
|
||||
| Machine | Unit | Fate |
|
||||
|---------|------|------|
|
||||
| Dankclaw | `frank-default-small` (FastFlowLM NPU, port 52625) | Disabled; unit file removed from `/etc/systemd/system`. The service keeps running until stopped or rebooted. |
|
||||
| Dankclaw | `frank-large-ling` (Ling trial) | Disabled. |
|
||||
| Dankclaw | `vllm-thinkingcap` | Disabled (already). |
|
||||
| Dankserver | `frank-default-small` (old llama.cpp 35B) | Disabled; unit file removed. |
|
||||
| Dankserver | `llamaserver`, `ollama`, `vllm-awq-qwen36`, `vllm-qwen36` | Disabled. |
|
||||
|
||||
`systemctl disable` removes the symlinked unit file from `/etc/systemd/system` when the file is a symlink. The repo copies stay intact. To restore a disabled service, re-symlink the unit and enable it:
|
||||
|
||||
```
|
||||
sudo ln -sf /home/dank/<repo>/scripts/<unit>.service /etc/systemd/system/<unit>.service
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl enable <unit>
|
||||
```
|
||||
|
||||
The NPU re-enable recipe uses the Dankclaw repo path `/home/dank/vllmScripts/scripts/frank-default-small.service`.
|
||||
|
||||
The Dankserver helper `frank-restart-small.sh` targets the removed `frank-default-small` unit. Do not use it. Restart the PP worker with an explicit command:
|
||||
|
||||
```
|
||||
sudo systemctl restart frank-pp-worker
|
||||
```
|
||||
|
||||
`scripts/deploy.sh dankserver` also targets the removed unit. Do not run it until the script is retargeted to `frank-pp-worker`. The `dankclaw` target works: it restarts the PP head through `frank-restart-large.sh`.
|
||||
|
||||
## Workflow
|
||||
|
||||
@@ -77,7 +109,7 @@ Each target has a root-owned helper script that performs the full restart. The u
|
||||
| Target | Helper | Sudoers drop-in |
|
||||
|--------|--------|-----------------|
|
||||
| Dankclaw | `/usr/local/sbin/frank-restart-large.sh` | `/etc/sudoers.d/frank-large` |
|
||||
| Dankserver | `/usr/local/sbin/frank-restart-small.sh` | `/etc/sudoers.d/frank-small` |
|
||||
| Dankserver | `/usr/local/sbin/frank-restart-small.sh` (obsolete, see below) | `/etc/sudoers.d/frank-small` |
|
||||
|
||||
The Dankclaw helper does these steps:
|
||||
|
||||
@@ -87,7 +119,7 @@ The Dankclaw helper does these steps:
|
||||
4. Reloads systemd.
|
||||
5. Enables and restarts the service.
|
||||
|
||||
The Dankserver helper does the same steps without the cache drop.
|
||||
The Dankserver helper `frank-restart-small.sh` is obsolete: it restarts the removed `frank-default-small` unit. Do not use it. The sudoers drop-in does not cover `frank-pp-worker`, so the worker restart needs the password (or a new helper). Restart the worker with `sudo systemctl restart frank-pp-worker`.
|
||||
|
||||
The sudoers drop-in contains exactly two rules:
|
||||
|
||||
@@ -125,17 +157,24 @@ curl -sf http://127.0.0.1:7770/health
|
||||
|
||||
### Update Dankserver
|
||||
|
||||
Run these commands on Dankserver:
|
||||
Run these commands on Dankserver to update and restart the PP worker:
|
||||
|
||||
```
|
||||
cd /home/dank/vLLM
|
||||
git pull origin main
|
||||
sudo ln -sf /home/dank/vLLM/scripts/frank-default-small.service /etc/systemd/system/frank-default-small.service
|
||||
sudo ln -sf /home/dank/vLLM/scripts/frank-pp-worker.service /etc/systemd/system/frank-pp-worker.service
|
||||
sudo systemctl daemon-reload
|
||||
sudo systemctl restart frank-default-small
|
||||
curl -sf http://127.0.0.1:7770/health
|
||||
sudo systemctl restart frank-pp-worker
|
||||
```
|
||||
|
||||
The worker has no HTTP health endpoint. Verify the listener instead:
|
||||
|
||||
```
|
||||
ss -tln | grep :50052
|
||||
```
|
||||
|
||||
The old passwordless helper `frank-restart-small.sh` targets the removed `frank-default-small` unit. Do not use it. See Boot Policy.
|
||||
|
||||
## Service Units
|
||||
|
||||
### Dankclaw — frank-default-large
|
||||
@@ -189,15 +228,9 @@ layer offload (`--gpu-layers all`, `--fit off`), batch 8192 / ubatch
|
||||
> table. Also retired earlier: the vLLM-ROCm CIRU INT4 Ling-3.0-flash
|
||||
> runtime, and the llama-serve/vLLM units on Dankclaw.
|
||||
|
||||
### Dankserver — frank-default-small (STOPPED 2026-08-16)
|
||||
### Dankserver — old llama.cpp 35B service (retired 2026-08-16)
|
||||
|
||||
> Note (2026-08-16): the `frank-default-small` alias now belongs to the FastFlowLM NPU endpoint on Dankclaw (port 52625, see [[FastFlowLM NPU Inference]]). The llama.cpp Qwen3.6-35B-A3B service on Dankserver described below is stopped (verified `inactive` 2026-08-16). The text below is kept as history.
|
||||
|
||||
The active service on Dankserver runs llama.cpp (upstream, `b10108+`) and serves the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` model under the alias `frank-default-small` on port 7770. It uses **ngram-mod** speculative decoding (`--spec-ngram-mod-n-min 48 / n_max 64 / n_match 24`), **4 parallel slots** at 175104 context each (pool 700416), and mmproj vision. **36 of 40 MoE expert layers** stay on CPU (`--n-cpu-moe 36`); 4 expert layers' weights are back on GPU alongside all non-expert tensors (`--gpu-layers all`). KV stays on GPU (K=`f16`, V=`f16`), refitted from 800000 on 2026-08-14 when the pool went dual-f16 (the f16/f16 pool costs 13680 MiB at 700416 cells and did not fit at 800768 cells). Batch 8192 / ubatch 3072, `--ctx-checkpoints 4`, `--no-mmap`. The unit file is `scripts/frank-default-small.service`, symlinked to `/etc/systemd/system/`. It launches through `scripts/serve-llama-frank-small.sh`, which reads tunables from `scripts/frank-default-small.env`.
|
||||
|
||||
> Note: the Dankserver process currently observed runs `--spec-type ngram-mod` with `--parallel 4`, batch 8192 / ubatch 2048, and `--n-cpu-moe 34` (this is being bench-tuned by the operator; reconcile against `scripts/frank-default-small.env`).
|
||||
|
||||
> Note: this service replaced the retired `vllm-awq-qwen36` (AWQ 4-bit, port 34401) and the earlier `vllm-qwen36` (FP8) / `llama-server` (ROCmFPX) units. The repo tracks `main` on Dankserver at `/home/dank/vLLM`.
|
||||
Dankserver previously ran `frank-default-small`: llama.cpp serving the HauhauCS Qwen3.6-35B-A3B uncensored `Q6_K_P` (ngram-mod spec, 4 slots x 175104 ctx, 36 of 40 MoE expert layers on CPU, mmproj vision). The service was verified `inactive` on 2026-08-16, and its unit file was removed on 2026-08-17 under the boot policy. The name `frank-default-small` now belongs to the FastFlowLM NPU endpoint on Dankclaw (see [[FastFlowLM NPU Inference]]). The RTX 3090 now runs the PP worker. See git history for the full retired configuration.
|
||||
|
||||
## Validation
|
||||
|
||||
@@ -230,17 +263,11 @@ If a service segfaults at load with no obvious cause, compare the exported env a
|
||||
|
||||
## Incident: Jobs "Released Prematurely" Under Parallel Load (2026-08-13)
|
||||
|
||||
The user reported that on `frank-default-large` (Dankclaw), a job that is streaming tokens gets released early when another job starts streaming. The suspicion was the speculative decoding configuration ("dspark"). Investigation found no dspark in either deployment: Dankclaw runs `draft-dflash`, Dankserver runs `ngram-mod`. Three findings emerged:
|
||||
A user reported that a streaming job on `frank-default-large` was released early when another job started. The suspicion was speculative decoding. The investigation found three facts:
|
||||
|
||||
1. **The server does not preempt.** `llama-server` defers a new task into a queue when all slots are busy. It never releases a running generation to make room. Verified empirically: two concurrent streaming jobs completed normally with dissimilar prompts, similar prompts, and near-identical prompts (high LCP reuse). `parallel=3` on Dankclaw serves up to three concurrent jobs and queues the rest.
|
||||
2. **The releases were client-side aborts.** The pi coding agent aborts a stream when the user interrupts it (for example, Escape, or starting a new message in a way that cancels the previous one). The client records `stopReason: "aborted"` with `errorMessage: "Operation aborted"`, and the server logs `srv stop: cancel task, id_task = N`. The server journal shows a `slot release` line with no `cancel task` line when the generation ends normally (EOS or limit).
|
||||
3. **Dankserver crashed under the parallel researcher load.** Five parallel research subagents hammered `frank-default-small` with back-to-back requests (a new task every few seconds for ~15 minutes). At 19:45:44 and again at 19:55:27 the service aborted with `CUDA error: an unsupported value or parameter was passed to the function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541`. systemd restarted it, and in-flight streams died with `Connection error.` or `terminated`, then `503 Loading model` during the ~5 minute reload. This crash, not speculative decoding, killed the parallel jobs. The signature is a CUBLAS invalid-value error in a GEMM under heavy concurrent load; root cause on Dankserver (llama.cpp b10121, driver 595.84, MoE experts CPU-offloaded) was not yet determined.
|
||||
|
||||
Diagnosis recipe: to tell a client abort from a server release, read the journal around the release time. A client abort shows `W srv stop: cancel task, id_task = N`. A natural stop shows only `slot release` with `truncated = 0` and the final timings line.
|
||||
|
||||
### VRAM and RAM growth under load
|
||||
|
||||
This crash class recurs on Dankserver. The 2026-08-14 investigation ruled out ctx-checkpoints and ngram-mod as VRAM growth sources and confirmed RAM growth. See [[dankserver-vram-growth]] for the full findings, the monitoring setup, and the mitigation options.
|
||||
1. **The server does not preempt.** `llama-server` defers a new task into a queue when all slots are busy. It never releases a running generation to make room. Verified empirically with concurrent streaming jobs.
|
||||
2. **The releases were client-side aborts.** The pi coding agent aborts a stream when the user interrupts it. The client records `stopReason: "aborted"`; the server logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`.
|
||||
3. **Dankserver crashed under the parallel researcher load.** Five parallel research subagents hammered `frank-default-small` with back-to-back requests. The service aborted twice with `CUDA error: an unsupported value or parameter was passed to the function` in `cublasSgemm_v2` at `ggml-cuda.cu:1541`, then `status=6/ABRT`, then `503 Loading model` for ~5 minutes. That crash, not speculative decoding, killed the parallel jobs. The affected service is retired; see [[dankserver-vram-growth]] for the VRAM/RAM growth findings behind the crash class.
|
||||
|
||||
## See Also
|
||||
|
||||
|
||||
+8
-6
@@ -19,21 +19,23 @@ frank-pp-worker, port 50052]
|
||||
D[gfx1151 APU 30GiB phys / 96GiB GPU-visible]
|
||||
E[llama.cpp PP head
|
||||
frank-default-large, port 7770]
|
||||
F[Qwen3.8-27B UD-Q8_K_XL
|
||||
split 0.40/0.60 to worker, MTP+ngram-mod]
|
||||
F[Qwen3.8-27B-Uncensored Q6_K
|
||||
split 0.68/0.32 to worker, standalone MTP draft + ngram-mod]
|
||||
D --> E --> F
|
||||
G[FastFlowLM NPU XDNA2 50 TOPS
|
||||
frank-default-small, port 52625]
|
||||
H[qwen3.6-moe:35b-a3b
|
||||
12.5-13.1 t/s, 262k ctx, vision]
|
||||
frank-default-small, port 52625 - DISABLED for boot 2026-08-17]
|
||||
H[qwythos:9b
|
||||
Q4NX 7.76 GB, 262k ctx]
|
||||
D --> G --> H
|
||||
end
|
||||
|
||||
I[LLM Clients / API] -->|HTTP :7770| E
|
||||
I -->|HTTP :52625| G
|
||||
E -->|RPC layers 0.40| B
|
||||
E -->|RPC layers 0.68| B
|
||||
```
|
||||
|
||||
Boot policy (2026-08-17): only `frank-default-large` and `frank-pp-worker` are enabled. The NPU endpoint and all retired units are disabled; see [[deployment]] for the table and re-enable recipes.
|
||||
|
||||
## Servers
|
||||
|
||||
| Server | GPU | VRAM / Memory | OS |
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
# Ling-3.0-flash Q6_K Trial on Dankclaw
|
||||
|
||||
> **RETIRED 2026-08-17.** The `frank-large-ling` unit is disabled under the boot policy. `frank-default-large` now runs the Qwen3.8-27B PP head, not the Qwen3.6-27B config described in the (obsolete) switch-back section below.
|
||||
|
||||
This page records the Ling-3.0-flash trial that temporarily replaced
|
||||
`frank-default-large` on Dankclaw. It replaces the Qwen3.6-27B service on
|
||||
port 7770 under the same API alias.
|
||||
@@ -83,18 +85,13 @@ B) switch back to Qwen3.6-27B (frank-default-large, MTP n_max 3).
|
||||
Passwordless management is installed:
|
||||
`/usr/local/sbin/frank-restart-ling.sh` and `/etc/sudoers.d/frank-ling`.
|
||||
|
||||
## Switch Back to Qwen3.6-27B (MTP)
|
||||
## Switch Back (obsolete)
|
||||
|
||||
The Qwen config was not changed. It remains committed as
|
||||
`frank-default-large.service` / `serve-llama-frank-large.sh` /
|
||||
`frank-default-large.env` (draft-mtp, n_max 3, ubatch 8192).
|
||||
|
||||
Run these commands:
|
||||
|
||||
```
|
||||
sudo systemctl stop frank-large-ling
|
||||
sudo systemctl start frank-default-large
|
||||
```
|
||||
The switch-back commands below are obsolete. `frank-default-large` now
|
||||
runs the Qwen3.8-27B pipeline-parallel head. To restore a Ling trial,
|
||||
re-enable `frank-large-ling` from the repo unit
|
||||
(`scripts/frank-large-ling.service`) and serve script
|
||||
(`scripts/serve-ling-flash.sh`), then start it on a non-conflicting port.
|
||||
|
||||
## Caveats
|
||||
|
||||
|
||||
@@ -1,40 +1,10 @@
|
||||
# Ling-3.0-flash on Dankclaw — vLLM-ROCm Deployment Plan
|
||||
# Ling-3.0-flash on Dankclaw — vLLM-ROCm Deployment Plan (never shipped)
|
||||
|
||||
## Decisions (confirmed)
|
||||
This page was the plan for a vLLM-ROCm deployment of
|
||||
`jcbtc/Ling-3.0-Flash-CIRU-int4-Strix-native` (INT4 W4A16, ~62 GB) on
|
||||
Dankclaw. The deployment never shipped. The actual Ling work used the
|
||||
llama.cpp Q6_K fork trial instead: [[ling-3.0-flash-trial]].
|
||||
|
||||
| Decision | Choice |
|
||||
|----------|--------|
|
||||
| Objective | Quality-first; correctness (tool calling) over raw quant speed |
|
||||
| Runtime | vLLM-ROCm on host ROCm venv (jcbtc validated gfx1151 path) |
|
||||
| Model | jcbtc/Ling-3.0-Flash-CIRU-int4-Strix-native (INT4 W4A16, ~62 GB) |
|
||||
| Accuracy | INT4 primary; MXFP4 (olka-fi) as controlled A/B after stability |
|
||||
| KV cache | Full precision (no FP8 KV) — quality |
|
||||
| Speculative decoding | MTP (native, lossless) |
|
||||
| Context | Native 256k; user cap ~200k; 4-6 concurrent sessions (~8 max_num_seqs) |
|
||||
| Throughput | Maximum (chunked prefill, large max_num_batched_tokens) |
|
||||
| systemd unit | frank-default-large.service |
|
||||
| Served alias | frank-default-large |
|
||||
| Port | 7770 (8080s overused) |
|
||||
| Migration | Kill llama.cpp, disable llama-server.service, reclaim memory |
|
||||
| Delivery | Follow AGENTS.md git workflow; scripts/ + docs/ in repo |
|
||||
|
||||
## Why not alternatives
|
||||
|
||||
- FP8: broken/lossy on Strix Halo; no ROCmFP8 Ling model; 124 GB does not fit.
|
||||
- HIGHER bpw (Q5/Q6/Q8/BF16/FP8): exceed 96 GB usable wall for a 124B MoE.
|
||||
- ROCmFP4 (raulvidis): defective tool calling; llama.cpp-only concurrency limits.
|
||||
- Vulkan: llama.cpp backend only, not vLLM; no quality ceiling unlock.
|
||||
- FP8 KV cache: rejected for quality (full-precision KV affordable because
|
||||
Ling is hybrid linear-attention with small KV footprint).
|
||||
|
||||
## Steps
|
||||
|
||||
1. Pre-flight: disk space, ROCm/amdgpu state, memory baseline.
|
||||
2. Bring jcbtc scripts + frank-default-large.service into repo; commit.
|
||||
3. Build ROCm vLLM (install-host-deps.sh + build-vllm-gfx1151.sh).
|
||||
4. Download weights via download-model.sh (~62 GB).
|
||||
5. Validate on staging: MTP engages, tool calling works, quality + throughput.
|
||||
6. Cut over: kill llama.cpp, disable llama-server, enable frank-default-large on 7770.
|
||||
7. Tune max throughput; confirm concurrent-token capacity.
|
||||
8. MXFP4 A/B (post-stable).
|
||||
9. Update AGENTS.md, README, and KB.
|
||||
The trial service (`frank-large-ling`) is retired and disabled under the
|
||||
boot policy (2026-08-17). `frank-default-large` is the Qwen3.8-27B PP
|
||||
head, not the Ling path. See git history for the full decision table.
|
||||
|
||||
+43
-117
@@ -1,9 +1,8 @@
|
||||
# Qwen3.8-27B UD-Q8_K_XL Speculative-Decoding Matrix
|
||||
# Qwen3.8-27B Speculative-Decoding Matrix
|
||||
|
||||
This page records the speculative-decoding benchmark and the production
|
||||
tuning for the unsloth Qwen3.8-27B `UD-Q8_K_XL` service
|
||||
(`frank-default-large`), now a pipeline-parallel head with a Dankserver
|
||||
worker.
|
||||
This page records the speculative-decoding benchmark history and the
|
||||
production tuning for the Qwen3.8-27B service (`frank-default-large`),
|
||||
now a pipeline-parallel head with a Dankserver worker.
|
||||
|
||||
## Production state (2026-08-17: Q6_K, q8_0/f16 KV, split 0.68)
|
||||
|
||||
@@ -42,99 +41,44 @@ Current production, verified against the live process and journal
|
||||
- Batch 8192 / ubatch 256, `--direct-io`, `--fit off`,
|
||||
`--gpu-layers all`.
|
||||
|
||||
## Result (superseded 2026-08-17: Q8_K_XL, f16/f16 KV, split 0.53)
|
||||
|
||||
Superseded for topology on 2026-08-16, when the PP split became
|
||||
production. The ngram-mod preset survives as the fallback layer under
|
||||
MTP. The Vulkan0 verdict in this section applies to the standalone
|
||||
configuration only.
|
||||
|
||||
- Model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB, bundled MTP).
|
||||
- Spec: llama.cpp **`--spec-default`** preset = **ngram-mod** (n_match 24,
|
||||
n_min 48, n_max 64). No draft model. Chosen 2026-08-15 after DSpark
|
||||
crashed under parallel load (see below).
|
||||
- **PARALLEL=2** — passes the 2x32k parallel stream test (both streams
|
||||
complete 256/256 concurrently, no crash). ngram-mod is draftless and
|
||||
parallel-safe.
|
||||
- **Backend: Vulkan0** — at 262144 ctx (~200k prompt) Vulkan decodes
|
||||
5.68 t/s vs ROCm 3.83 t/s (48% faster); ROCm only wins prefill (119 vs
|
||||
106 t/s). See the high-context bench table below.
|
||||
- Batch 8192 / ubatch 256, ctx 524288 (262144 per slot).
|
||||
|
||||
## Result (superseded 2026-08-15: DSpark)
|
||||
|
||||
- Model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB, bundled MTP).
|
||||
- Spec: **DSpark sidecar draft** `magnitudedev/Qwen3.8-27B-DSpark-GGUF`
|
||||
(`Qwen3.8-27B-DSpark-Q8_0.gguf`, 1.36B, block_size 7), `--spec-type
|
||||
draft-dspark`, `--spec-draft-n-max 7` (clamps to block_size).
|
||||
- **PARALLEL=1.** Two concurrent 16k prose prompts crashed the ROCm backend
|
||||
with `ROCm error: an illegal memory access` during per-sequence block
|
||||
sampling. See the DSpark parallel incident below.
|
||||
- Retired the same day in favor of `--spec-default`.
|
||||
- Batch 8192 / ubatch 256, ctx 524288 (262144 per slot).
|
||||
|
||||
## Result (superseded 2026-08-14: bundled MTP)
|
||||
|
||||
- Model: `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB, bundled MTP).
|
||||
- Spec: `draft-mtp`, `--spec-draft-n-max 5`, no `--spec-draft-p-min`.
|
||||
- Batch 8192 / ubatch 256, parallel 2, ctx 524288 (262144 per slot).
|
||||
- Realistic-prompt generation: ~13.3 t/s single, ~8 t/s per slot under
|
||||
2 concurrent streams. No premature stop.
|
||||
|
||||
## Incident: DSpark ROCm Crash Under Parallel Load (2026-08-15)
|
||||
|
||||
A two-stream parallel acceptance test crashed `frank-default-large`.
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
A[Two concurrent 16k prose prompts<br>high prompt-cache reuse] --> B[ROCm illegal memory access<br>in common_speculative_impl_draft_dflash::draft]
|
||||
B --> C[status 6/ABRT core dump]
|
||||
C --> D[systemd auto-restart, 503 during reload]
|
||||
D --> E[pinned PARALLEL=1]
|
||||
```
|
||||
|
||||
Trigger: two distinct-but-overlapping ~16k-token realistic prose prompts
|
||||
started simultaneously on PARALLEL=2. The ROCm backend aborted at
|
||||
`ggml-cuda.cu:106` with `ROCm error: an illegal memory access was
|
||||
encountered`, surfacing in `hipStreamSynchronize` inside
|
||||
`common_speculative_impl_draft_dflash::draft` (the class used by
|
||||
`draft-dspark`). `systemd` caught it (status 6/ABRT) and restarted the
|
||||
unit; in-flight streams saw `Remote end closed connection without response`
|
||||
until the ~reload finished.
|
||||
|
||||
Mechanism: DSpark reuses the DFlash block-diffusion draft machinery. The
|
||||
DFlash draft model uses one shared draft context (`ctx_dft`) across all
|
||||
parallel slots; under concurrent block drafting on ROCm it performs an
|
||||
illegal memory access (HIP stream sync). This is the same multi-slot
|
||||
class of bug that retired `draft-dflash` for premature EOS; here it is a
|
||||
hard ROCm crash instead of a bad sample.
|
||||
|
||||
Fix: pin **PARALLEL=1** (single slot). With one slot, concurrent requests
|
||||
queue safely and each completes to `max_tokens`. Revisit after an upstream
|
||||
multi-slot DSpark fix (DSpark is known-good single-slot). Single-slot 16k
|
||||
prose generation was ~steady (measuring only the correct-size signal, not
|
||||
this page).
|
||||
|
||||
## Parallel results by stream count (2026-08-15, correct-size tests)
|
||||
|
||||
| Parallel | Two 16k prose prompts | Two 67k prose prompts |
|
||||
| -------- | -------------------- | --------------------- |
|
||||
| 2 | crashes (illegal mem access) | passes |
|
||||
| 1 | queues, both complete | n/a |
|
||||
|
||||
Note: the crash manifests on the smaller (16k) concurrent streams, not the
|
||||
larger 67k pair. Size is not the trigger; concurrent block drafting on the
|
||||
shared draft context is.
|
||||
|
||||
## How MTP works on Qwen3.8
|
||||
## How MTP works on Qwen3.8 (bundled-head era)
|
||||
|
||||
The unsloth `UD-*` GGUFs keep the MTP (Multi-Token Prediction) head inside
|
||||
the model file. llama.cpp b10411+ auto-detects it and creates the draft
|
||||
context from the target model. Do not pass `--spec-draft-model`.
|
||||
context from the target model. Do not pass `--spec-draft-model` for those
|
||||
files. This is superseded: the production `Qwen3.8-27B-Uncensored-Q6_K.gguf`
|
||||
has NO bundled head, and the serve script passes the standalone draft
|
||||
sidecar with `--spec-draft-model` (see Production state above).
|
||||
|
||||
The stock llama.cpp build must be b10411 or newer. The 2026-08-14 rebuild
|
||||
on Dankclaw is b10433 (`9b05354ec`) with `GGML_HIP` + `gfx1151` + Vulkan.
|
||||
|
||||
## History (superseded configurations, 2026-08-14 to 08-16)
|
||||
|
||||
All superseded configs used `Qwen3.8-27B-UD-Q8_K_XL.gguf` (29.29 GiB,
|
||||
bundled MTP) before the Q6_K swap. The ngram-mod preset survives as the
|
||||
fallback layer under MTP. The Vulkan0 verdict from this era applies to the
|
||||
standalone configuration only; the PP head runs ROCm0.
|
||||
|
||||
- **2026-08-14: bundled MTP** — `draft-mtp`, n_max 5, parallel 2, ctx
|
||||
524288 (2 x 262144). ~13.3 t/s single, ~8 t/s per slot under 2
|
||||
concurrent streams. No premature stop.
|
||||
- **2026-08-15: DSpark sidecar** — `magnitudedev/Qwen3.8-27B-DSpark-GGUF`
|
||||
(Q8_0, 1.36B, block_size 7), n_max 7, PARALLEL=1. Two concurrent 16k
|
||||
prose prompts crashed the ROCm backend with `ROCm error: an illegal
|
||||
memory access` in `common_speculative_impl_draft_dflash::draft` (status
|
||||
6/ABRT, systemd restart, ~503 during reload). DSpark reuses the DFlash
|
||||
block-diffusion machinery, which shares one draft context across
|
||||
parallel slots; concurrent block drafting faults in HIP stream sync.
|
||||
This is the same multi-slot bug class that retired `draft-dflash`
|
||||
(premature EOS). The crash shows on 16k concurrent streams, not on a
|
||||
67k pair — concurrency is the trigger, not size. Fix: PARALLEL=1
|
||||
(requests queue safely). Retired the same day.
|
||||
- **2026-08-15 to 08-16: `--spec-default` (ngram-mod), Vulkan0** — n_match
|
||||
24 / n_min 48 / n_max 64, draftless, PARALLEL=2 (2 x 32k parallel stream
|
||||
test passes). Chosen for parallel safety after DSpark; kept as the
|
||||
fallback under MTP in the current config.
|
||||
|
||||
## Benchmark Method
|
||||
|
||||
The matrix ran 54 configurations: 3 spec types (`draft-mtp,ngram-mod`,
|
||||
@@ -189,37 +133,19 @@ ubatch 256, single slot, on scratch port 7892.
|
||||
|
||||
Verdict: at very high context, Vulkan0 decodes **48% faster** (5.68 vs
|
||||
3.83 t/s); ROCm0 prefills 12% faster. Generation is the interactive cost
|
||||
(users stream tokens), so **Vulkan0** is the production backend. Both
|
||||
`tg_3s` values were rock-steady across the run (3.83 and 5.67 t/s).
|
||||
(users stream tokens). This verdict applies to the standalone
|
||||
configuration; the PP head runs ROCm0 (HIP registers before Vulkan).
|
||||
Both `tg_3s` values were rock-steady across the run (3.83 and 5.67 t/s).
|
||||
High-context decode collapses on both backends versus ~25 t/s at 16k ctx;
|
||||
that is the long-context attention penalty, not a spec artifact.
|
||||
|
||||
## Production Configuration
|
||||
|
||||
`scripts/frank-default-large.env` (live 2026-08-16):
|
||||
|
||||
```ini
|
||||
RPC_SERVERS=192.168.0.235:50052 # Dankserver worker
|
||||
TENSOR_SPLIT=0.40,0.60 # [RPC0, iGPU]
|
||||
CTX_SIZE=401400 # pool = 2 slots x 200700
|
||||
PARALLEL=2
|
||||
SPEC_NONE=0
|
||||
SPEC_DEFAULT=1 # ngram-mod fallback (48/64/24)
|
||||
MTP=1 # draft-mtp, bundled head
|
||||
SPEC_DRAFT_N_MAX=4
|
||||
SPEC_NGRAM_MOD_N_MIN=48
|
||||
SPEC_NGRAM_MOD_N_MAX=64
|
||||
SPEC_NGRAM_MOD_N_MATCH=24
|
||||
CACHE_TYPE_K=f16
|
||||
CACHE_TYPE_V=f16
|
||||
BATCH_SIZE=8192
|
||||
UBATCH_SIZE=256
|
||||
```
|
||||
|
||||
There is no `DEVICE` variable. The head must not pass `--device` (the RPC
|
||||
device would be silently dropped). Live slot context: `n_ctx_slot =
|
||||
200704`. The pi `models.json` contextWindow for `frank-default-large`
|
||||
declares 200000, a safe under-estimate of the live slot.
|
||||
The live tunables live in `scripts/frank-default-large.env` (head) and
|
||||
`scripts/frank-pp-worker.env` (worker) in this repo. The Production
|
||||
state section at the top of this page records the current values: Q6_K
|
||||
model + standalone MTP draft, split 0.68/0.32, 262144 ctx, PARALLEL=1,
|
||||
q8_0/f16 KV.
|
||||
|
||||
## See Also
|
||||
|
||||
|
||||
Reference in New Issue
Block a user