Swap frank-default-large to DFlash2 on Vulkan (PR #27342)
Replace the retired CIRU ROCmFPX runtime with upstream llama.cpp PR #27342 DFlash2 on the Vulkan backend: - Target: unsloth Qwen3.8-27B-UD-Q8_K_XL.gguf (31.4 GB, already on disk) - Drafter: z-lab Qwen3.8-27B-DFlash2-Q4_K_M.gguf (block size 8, top-k 16) - Runtime: /home/dank/llama.cpp-dflash2/build-dflash2/bin/llama-server (GGML_VULKAN=ON; HIP lacks TOP_K so the stack runs on Vulkan0/RADV) - spec: --spec-type draft-dflash --spec-draft-n-max 8 (clamps to 7) Validated 2026-08-21 on scratch port 52626: mean acceptance 5.27, health + reasoning completion OK. Archive the retired CIRU .env/.serve script in .AGENTTRASH. Update deployment.md, index.md, AGENTS.md; add docs/dflash2-large-vulkan.md.
This commit is contained in:
Frank committed
1 parent
701e742f9f
commit
fd0d9dd3cb
9 files changed
+470
-136
No files matched your search
@@ -3,23 +3,21 @@
|
||||
## Overview
|
||||
|
||||
This project manages deployment scripts for the LLM inference fleet on two
|
||||
local machines. Topology (2026-08-18):
|
||||
local machines. Topology (2026-08-21):
|
||||
|
||||
| Target | Host | Runtime | Service | Model | Port |
|
||||
|--------|------|---------|---------|-------|------|
|
||||
| Dankserver | dank@dankserver | llama.cpp (upstream b10121) | frank-default-small | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU | 7770 |
|
||||
| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (ROCm 7.15 SDK in the ling3-ciru venv) | frank-default-large | jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP, PromptForge DualView, 262K context / 1 slot | 52625 |
|
||||
| Dankclaw | dank@dankclaw | llama.cpp PR #27342 DFlash2 (Vulkan RADV) | frank-default-large | unsloth Qwen3.8-27B-UD-Q8_K_XL + z-lab Qwen3.8-27B-DFlash2-Q4_K_M drafter, 262K context / 1 slot | 52625 |
|
||||
|
||||
The 2026-08-17 redo retired the pipeline-parallel split and the FastFlowLM
|
||||
NPU endpoint. The 2026-08-18 swap retired the vLLM service on Dankclaw; the
|
||||
CIRU ROCmFPX llama.cpp runtime now runs on the Strix Halo iGPU (gfx1151).
|
||||
A second 2026-08-18 swap renamed the services and restored the Qwen3.6 35B
|
||||
A3B Q6_K_P model on Dankserver: Dankserver is now `frank-default-small`
|
||||
(Qwen3.6-35B-A3B Q6_K_P + ngram-mod, MoE experts on CPU) and Dankclaw is
|
||||
now `frank-default-large` (CIRU Qwen3.8-27B, 262K context / 1 slot). The
|
||||
retired Qwen3.8-27B IQ4_XS FastMTP files live in `.AGENTTRASH/`. See
|
||||
docs/ciru-activefpx-promptforge.md. The old scripts live in `old/scripts/`
|
||||
for reference; they describe the retired PP topology.
|
||||
NPU endpoint. The 2026-08-18 swap retired the vLLM service and the CIRU
|
||||
ROCmFPX runtime on Dankclaw. On 2026-08-21 `frank-default-large` moved to
|
||||
upstream llama.cpp PR #27342 with DFlash2 speculative decoding on the
|
||||
Vulkan backend (`Vulkan0`, RADV gfx1151). The retired CIRU AI and IQ4_XS
|
||||
files live in `.AGENTTRASH/`. See docs/dflash2-large-vulkan.md. The old
|
||||
scripts live in `old/scripts/` for reference; they describe the retired PP
|
||||
topology.
|
||||
|
||||
**Sibling project:** [../serverAppConfigs](../serverAppConfigs) manages the
|
||||
Podman app stack on the same hosts (OpenWebUI, Gitea, observability). Its
|
||||
@@ -65,7 +63,7 @@ Use `scripts/deploy.sh` to deploy committed changes to a target. It pushes `main
|
||||
|
||||
```
|
||||
FRANK_SUDO_PW=... ./scripts/deploy.sh dankserver # llama.cpp FastMTP, port 7770
|
||||
FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw # CIRU llama.cpp, port 52625
|
||||
FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw # DFlash2 llama.cpp, port 52625
|
||||
./scripts/deploy.sh dankserver --dry-run # preview without changes
|
||||
```
|
||||
|
||||
@@ -127,23 +125,19 @@ back to a slow path and prefill collapses ~12x (measured 2026-08-17: 95 t/s
|
||||
vs 1300+ t/s at an 8K prompt on the 3090). See the HauhauCS HF repo README
|
||||
for the base recipe.
|
||||
|
||||
### CIRU ROCmFPX llama.cpp (Dankclaw)
|
||||
### DFlash2 llama.cpp (Dankclaw, Vulkan)
|
||||
|
||||
The CIRU llama-server serves `jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge`
|
||||
from `/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server`.
|
||||
The runtime is the ciru-ai/ROCmFPX tag `qwen3.8-activefpx-promptforge-v2.3`
|
||||
(commit `103af8b41`) built against the private TheRock ROCm 7.15 SDK kept
|
||||
in the retired ling3-ciru venv
|
||||
(`.venv/lib/python3.12/site-packages/_rocm_sdk_*`). The model card requires
|
||||
ROCm 7.15 for the advertised Strix Halo performance; the system ROCm 7.2.4
|
||||
selects slower kernels. The serve script exports the 7.15 SDK libraries
|
||||
first in `LD_LIBRARY_PATH`.
|
||||
The upstream llama.cpp PR #27342 llama-server serves
|
||||
`unsloth/Qwen3.8-27B-UD-Q8_K_XL.gguf` with a z-lab DFlash2 drafter
|
||||
(`Qwen3.8-27B-DFlash2-Q4_K_M.gguf`) on the Vulkan backend. The binary lives
|
||||
at `/home/dank/llama.cpp-dflash2/build-dflash2/bin/llama-server`. DFlash2
|
||||
needs `TOP_K` + `ARGMAX`, which HIP does not implement, so the stack runs on
|
||||
`Vulkan0` (RADV, gfx1151). See docs/dflash2-large-vulkan.md.
|
||||
|
||||
The model files (v2 GGUF + FFN.pfs + GDN.pfs, ~34.9 GiB) live in
|
||||
`/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX/`. The full build runbook is
|
||||
in docs/ciru-activefpx-promptforge.md. The retired vLLM venvs
|
||||
(`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to
|
||||
`/home/dank/.AGENTTRASH/` on 2026-08-18.
|
||||
The target and drafter live in `/home/dank/models/`. The retired CIRU
|
||||
runtime (ROCmFPX v2.3, ROCm 7.15 SDK in the ling3-ciru venv, FFN/GDN
|
||||
companions) is archived in `.AGENTTRASH/` and documented in
|
||||
circu-activefpx-promptforge.md.
|
||||
|
||||
**Env var naming rule still applies:** the HIP runtime reads `TEMP`, `HOST`,
|
||||
and `GPU_MAX_HEAP_SIZE`. The serve script passes `--host`/`--port`/`--temp`
|
||||
@@ -192,7 +186,7 @@ graph TD
|
||||
|
||||
- `sudo` on both boxes needs a tty. Remote sudo steps fail non-interactively. Use `echo '<password>' | sudo -S <command>`.
|
||||
- Never run `pkill -f` with a pattern that appears in your own command line. It kills your ssh session. Use `pkill -x <exact-name>`.
|
||||
- Model load is slow: Dankserver loads the 30.6 GB Q6_K_P through RAM (`--no-mmap`, MoE experts on CPU) in a few minutes cold; Dankclaw CIRU loads the 34.9 GiB ActiveFPX payload in ~5-10 minutes. Health returns 503 while loading. Do not kill the process. `deploy.sh` waits up to 20 minutes on both targets (see `HEALTH_WAIT`).
|
||||
- Model load is slow: Dankserver loads the 30.6 GB Q6_K_P through RAM (`--no-mmap`, MoE experts on CPU) in a few minutes cold; Dankclaw DFlash2 loads the 31.4 GB UD-Q8_K_XL + 2.0 GB drafter in ~1 minute with a warm OS page cache and a few minutes cold. Health returns 503 while loading. Do not kill the process. `deploy.sh` waits up to 20 minutes on both targets (see `HEALTH_WAIT`).
|
||||
- A client abort and a server release look different in the journal. A client abort logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in [docs/deployment.md](docs/deployment.md).
|
||||
- Retired hardware history: the Dankserver 35B cuBLAS crash and the DFlash premature-EOS bug are historical; see [docs/dankserver-vram-growth.md](docs/dankserver-vram-growth.md) and [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md). The FastFlowLM NPU endpoint is retired; see [docs/FastFlowLM NPU Inference.md](docs/FastFlowLM%20NPU%20Inference.md).
|
||||
|
||||
|
||||
Reference in new issue
Block a user