dankclaw: replace vLLM with CIRU ROCmFPX llama.cpp runtime
Retire the vLLM service (frank-default-small, port 52625) on Dankclaw and serve jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge through the CIRU ROCmFPX llama.cpp runtime instead. - Build llama-server from ciru-ai/ROCmFPX tag qwen3.8-activefpx-promptforge-v2.3 (commit 103af8b41) against the private TheRock ROCm 7.15 SDK in the retired ling3-ciru venv. The model card requires ROCm 7.15 for the advertised Strix Halo performance; the system ROCm 7.2.4 selects slower kernels. CK pinned at fdf4bb7fcc984811cef48ce817d89aac064b984a; generated ck/config.h + ck/version.h copied into the CK source tree so the ggml-hip build can include them. LLAMA_BUILD_WEBUI=OFF (HF bucket llama-ui returns 404; server is consumed via the OpenAI API). - New scripts/serve-frank-small.sh + frank-default-small.env: 64K ctx, one slot, f16 KV, context checkpoints, prompt cache, native depth-4 MTP on ROCm0, PromptForge DualView sidecars (FFN + GDN), temp 0.7 top-p 0.95, reasoning off. LD_LIBRARY_PATH puts the 7.15 SDK libs before any system ROCm. - Update the systemd unit description (same service name and port). - deploy.sh: update the dankclaw target comment. - Docs: new docs/ciru-activefpx-promptforge.md with the build runbook; update docs/index.md and AGENTS.md topology. - The vLLM venvs (vllm-venv, vllm-vk-venv) were moved to /home/dank/.AGENTTRASH/ on the target; the vllm-vulkan plugin and docs were removed from the repo in an earlier commit.
This commit is contained in:
Frank committed
1 parent
4941d911f2
commit
3b51b5a94a
7 files changed
+359
-157
No files matched your search
@@ -3,16 +3,18 @@
|
||||
## Overview
|
||||
|
||||
This project manages deployment scripts for the LLM inference fleet on two
|
||||
local machines. Topology (2026-08-17 redo):
|
||||
local machines. Topology (2026-08-18):
|
||||
|
||||
| Target | Host | Runtime | Service | Model | Port |
|
||||
|--------|------|---------|---------|-------|------|
|
||||
| Dankserver | dank@dankserver | llama.cpp (patched FastMTP build) | frank-default-large | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS + FastMTP 32K draft + ngram-mod | 7770 |
|
||||
| Dankclaw | dank@dankclaw | vLLM (ROCm, /home/dank/vllm-venv) | frank-default-small | nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8, ngram spec, tool calling | 52625 |
|
||||
| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (ROCm 7.15 SDK in the ling3-ciru venv) | frank-default-small | jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP, PromptForge DualView | 52625 |
|
||||
|
||||
The 2026-08-17 redo retired the pipeline-parallel split and the FastFlowLM
|
||||
NPU endpoint. The RTX 3090 now runs the whole 27B model standalone
|
||||
(no RPC worker). Dankclaw runs vLLM on the Strix Halo iGPU (gfx1151).
|
||||
NPU endpoint. The RTX 3090 runs the whole 27B model standalone
|
||||
(no RPC worker). The 2026-08-18 swap retired the vLLM service on Dankclaw;
|
||||
frank-default-small now runs the CIRU ROCmFPX llama.cpp runtime on the
|
||||
Strix Halo iGPU (gfx1151). See docs/ciru-activefpx-promptforge.md.
|
||||
The old scripts live in `old/scripts/` for reference; they describe the
|
||||
retired PP topology.
|
||||
|
||||
@@ -60,7 +62,7 @@ Use `scripts/deploy.sh` to deploy committed changes to a target. It pushes `main
|
||||
|
||||
```
|
||||
FRANK_SUDO_PW=... ./scripts/deploy.sh dankserver # llama.cpp FastMTP, port 7770
|
||||
FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw # vLLM, port 52625
|
||||
FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw # CIRU llama.cpp, port 52625
|
||||
./scripts/deploy.sh dankserver --dry-run # preview without changes
|
||||
```
|
||||
|
||||
@@ -96,37 +98,27 @@ back to a slow path and prefill collapses ~12x (measured 2026-08-17: 95 t/s
|
||||
vs 1300+ t/s at an 8K prompt on the 3090). See the HauhauCS HF repo README
|
||||
for the base recipe.
|
||||
|
||||
### vLLM ROCm install (Dankclaw)
|
||||
### CIRU ROCmFPX llama.cpp (Dankclaw)
|
||||
|
||||
vLLM runs from `/home/dank/vllm-venv` (Python 3.12). Install recipe:
|
||||
The CIRU llama-server serves `jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge`
|
||||
from `/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server`.
|
||||
The runtime is the ciru-ai/ROCmFPX tag `qwen3.8-activefpx-promptforge-v2.3`
|
||||
(commit `103af8b41`) built against the private TheRock ROCm 7.15 SDK kept
|
||||
in the retired ling3-ciru venv
|
||||
(`.venv/lib/python3.12/site-packages/_rocm_sdk_*`). The model card requires
|
||||
ROCm 7.15 for the advertised Strix Halo performance; the system ROCm 7.2.4
|
||||
selects slower kernels. The serve script exports the 7.15 SDK libraries
|
||||
first in `LD_LIBRARY_PATH`.
|
||||
|
||||
```
|
||||
uv venv --python /usr/bin/python3.12 /home/dank/vllm-venv
|
||||
uv pip install --python /home/dank/vllm-venv vllm --torch-backend=auto
|
||||
```
|
||||
The model files (v2 GGUF + FFN.pfs + GDN.pfs, ~34.9 GiB) live in
|
||||
`/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX/`. The full build runbook is
|
||||
in docs/ciru-activefpx-promptforge.md. The retired vLLM venvs
|
||||
(`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to
|
||||
`/home/dank/.AGENTTRASH/` on 2026-08-18.
|
||||
|
||||
`--torch-backend=auto` resolves the AMD ROCm wheel set (torch +
|
||||
amd-torch-device-gfx1151 + the vllm rocm variant) from the AMD index.
|
||||
The model cache is the default HF cache
|
||||
(`/home/dank/.cache/huggingface`); the HF token lives at
|
||||
`~/.cache/huggingface/token`. The retired CIRU package
|
||||
(`/home/dank/ling3-ciru`) documents a source-build fallback for gfx1151.
|
||||
|
||||
**ROCm flash-attention requirement:** the vision encoder needs the Triton
|
||||
flash-attention path. Set `FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE` (in
|
||||
`frank-default-small.env`) and keep the pure-Python `flash_attn` package
|
||||
(copied from the CIRU venv) plus a `flash_attn_2_cuda` stub in the venv so
|
||||
`flash_attn.flash_attn_triton_amd` is findable. Without it the mm encoder
|
||||
falls back to torch SDPA and engine init OOMs (256 GiB score matrix).
|
||||
|
||||
**Tool calling (frank-default-small):** the server runs
|
||||
`--enable-auto-tool-choice --tool-call-parser qwen3_xml` with a custom
|
||||
chat template (`/home/dank/models/qwen36-chat_template-tools.jinja`).
|
||||
The template adds a concrete worked tool-call example and defaults tool
|
||||
requests to non-thinking (Qwen3 tool calls only work reliably with
|
||||
thinking off; normal requests keep thinking). The Avesed INT8 quant
|
||||
emitted malformed Qwen3 XML tool calls; the Quark W8A8 model follows the
|
||||
format cleanly.
|
||||
**Env var naming rule still applies:** the HIP runtime reads `TEMP`, `HOST`,
|
||||
and `GPU_MAX_HEAP_SIZE`. The serve script passes `--host`/`--port`/`--temp`
|
||||
as CLI flags only.
|
||||
|
||||
### Env var naming rule — HIP runtime collision
|
||||
|
||||
@@ -171,7 +163,7 @@ graph TD
|
||||
|
||||
- `sudo` on both boxes needs a tty. Remote sudo steps fail non-interactively. Use `echo '<password>' | sudo -S <command>`.
|
||||
- Never run `pkill -f` with a pattern that appears in your own command line. It kills your ssh session. Use `pkill -x <exact-name>`.
|
||||
- Model load is slow: Dankserver ~5 minutes cold (15.7 GB IQ4_XS through RAM, `--no-mmap`); Dankclaw vLLM ~5-10 minutes (38 GB checkpoint). Health returns 503 while loading. Do not kill the process. `deploy.sh` waits up to 10 minutes on Dankserver and 20 minutes on Dankclaw (see `HEALTH_WAIT`).
|
||||
- Model load is slow: Dankserver ~5 minutes cold (15.7 GB IQ4_XS through RAM, `--no-mmap`); Dankclaw CIRU ~5-10 minutes (34.9 GiB ActiveFPX payload). Health returns 503 while loading. Do not kill the process. `deploy.sh` waits up to 10 minutes on Dankserver and 20 minutes on Dankclaw (see `HEALTH_WAIT`).
|
||||
- A client abort and a server release look different in the journal. A client abort logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in [docs/deployment.md](docs/deployment.md).
|
||||
- Retired hardware history: the Dankserver 35B cuBLAS crash and the DFlash premature-EOS bug are historical; see [docs/dankserver-vram-growth.md](docs/dankserver-vram-growth.md) and [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md). The FastFlowLM NPU endpoint is retired; see [docs/FastFlowLM NPU Inference.md](docs/FastFlowLM%20NPU%20Inference.md).
|
||||
|
||||
|
||||
Reference in new issue
Block a user