dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

dankclaw: replace vLLM with CIRU ROCmFPX llama.cpp runtime

Browse Source 
Retire the vLLM service (frank-default-small, port 52625) on Dankclaw
and serve jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge through the CIRU
ROCmFPX llama.cpp runtime instead.

- Build llama-server from ciru-ai/ROCmFPX tag
  qwen3.8-activefpx-promptforge-v2.3 (commit 103af8b41) against the
  private TheRock ROCm 7.15 SDK in the retired ling3-ciru venv. The
  model card requires ROCm 7.15 for the advertised Strix Halo
  performance; the system ROCm 7.2.4 selects slower kernels.
  CK pinned at fdf4bb7fcc984811cef48ce817d89aac064b984a; generated
  ck/config.h + ck/version.h copied into the CK source tree so the
  ggml-hip build can include them. LLAMA_BUILD_WEBUI=OFF (HF bucket
  llama-ui returns 404; server is consumed via the OpenAI API).
- New scripts/serve-frank-small.sh + frank-default-small.env: 64K ctx,
  one slot, f16 KV, context checkpoints, prompt cache, native depth-4
  MTP on ROCm0, PromptForge DualView sidecars (FFN + GDN), temp 0.7
  top-p 0.95, reasoning off. LD_LIBRARY_PATH puts the 7.15 SDK libs
  before any system ROCm.
- Update the systemd unit description (same service name and port).
- deploy.sh: update the dankclaw target comment.
- Docs: new docs/ciru-activefpx-promptforge.md with the build runbook;
  update docs/index.md and AGENTS.md topology.
- The vLLM venvs (vllm-venv, vllm-vk-venv) were moved to
  /home/dank/.AGENTTRASH/ on the target; the vllm-vulkan plugin and
  docs were removed from the repo in an earlier commit.
This commit is contained in:
Frank committed 2026-08-17 21:12:35 -07:00
1 parent 4941d911f2
commit 3b51b5a94a
7 files changed
+359 -157
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

AGENTS.md
+26 -34
View File
@@ -3,16 +3,18 @@
## Overview ## Overview
This project manages deployment scripts for the LLM inference fleet on two This project manages deployment scripts for the LLM inference fleet on two
local machines. Topology (2026-08-17 redo): local machines. Topology (2026-08-18):
| Target | Host | Runtime | Service | Model | Port | | Target | Host | Runtime | Service | Model | Port |
|--------|------|---------|---------|-------|------| |--------|------|---------|---------|-------|------|
| Dankserver | dank@dankserver | llama.cpp (patched FastMTP build) | frank-default-large | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS + FastMTP 32K draft + ngram-mod | 7770 | | Dankserver | dank@dankserver | llama.cpp (patched FastMTP build) | frank-default-large | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS + FastMTP 32K draft + ngram-mod | 7770 |
| Dankclaw | dank@dankclaw | vLLM (ROCm, /home/dank/vllm-venv) | frank-default-small | nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8, ngram spec, tool calling | 52625 | | Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (ROCm 7.15 SDK in the ling3-ciru venv) | frank-default-small | jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP, PromptForge DualView | 52625 |
The 2026-08-17 redo retired the pipeline-parallel split and the FastFlowLM The 2026-08-17 redo retired the pipeline-parallel split and the FastFlowLM
NPU endpoint. The RTX 3090 now runs the whole 27B model standalone NPU endpoint. The RTX 3090 runs the whole 27B model standalone
(no RPC worker). Dankclaw runs vLLM on the Strix Halo iGPU (gfx1151). (no RPC worker). The 2026-08-18 swap retired the vLLM service on Dankclaw;
frank-default-small now runs the CIRU ROCmFPX llama.cpp runtime on the
Strix Halo iGPU (gfx1151). See docs/ciru-activefpx-promptforge.md.
The old scripts live in `old/scripts/` for reference; they describe the The old scripts live in `old/scripts/` for reference; they describe the
retired PP topology. retired PP topology.
@@ -60,7 +62,7 @@ Use `scripts/deploy.sh` to deploy committed changes to a target. It pushes `main
``` ```
FRANK_SUDO_PW=... ./scripts/deploy.sh dankserver # llama.cpp FastMTP, port 7770 FRANK_SUDO_PW=... ./scripts/deploy.sh dankserver # llama.cpp FastMTP, port 7770
FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw # vLLM, port 52625 FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw # CIRU llama.cpp, port 52625
./scripts/deploy.sh dankserver --dry-run # preview without changes ./scripts/deploy.sh dankserver --dry-run # preview without changes
``` ```
@@ -96,37 +98,27 @@ back to a slow path and prefill collapses ~12x (measured 2026-08-17: 95 t/s
vs 1300+ t/s at an 8K prompt on the 3090). See the HauhauCS HF repo README vs 1300+ t/s at an 8K prompt on the 3090). See the HauhauCS HF repo README
for the base recipe. for the base recipe.
### vLLM ROCm install (Dankclaw) ### CIRU ROCmFPX llama.cpp (Dankclaw)
vLLM runs from `/home/dank/vllm-venv` (Python 3.12). Install recipe: The CIRU llama-server serves `jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge`
from `/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server`.
The runtime is the ciru-ai/ROCmFPX tag `qwen3.8-activefpx-promptforge-v2.3`
(commit `103af8b41`) built against the private TheRock ROCm 7.15 SDK kept
in the retired ling3-ciru venv
(`.venv/lib/python3.12/site-packages/_rocm_sdk_*`). The model card requires
ROCm 7.15 for the advertised Strix Halo performance; the system ROCm 7.2.4
selects slower kernels. The serve script exports the 7.15 SDK libraries
first in `LD_LIBRARY_PATH`.
``` The model files (v2 GGUF + FFN.pfs + GDN.pfs, ~34.9 GiB) live in
uv venv --python /usr/bin/python3.12 /home/dank/vllm-venv `/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX/`. The full build runbook is
uv pip install --python /home/dank/vllm-venv vllm --torch-backend=auto in docs/ciru-activefpx-promptforge.md. The retired vLLM venvs
``` (`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to
`/home/dank/.AGENTTRASH/` on 2026-08-18.
`--torch-backend=auto` resolves the AMD ROCm wheel set (torch + **Env var naming rule still applies:** the HIP runtime reads `TEMP`, `HOST`,
amd-torch-device-gfx1151 + the vllm rocm variant) from the AMD index. and `GPU_MAX_HEAP_SIZE`. The serve script passes `--host`/`--port`/`--temp`
The model cache is the default HF cache as CLI flags only.
(`/home/dank/.cache/huggingface`); the HF token lives at
`~/.cache/huggingface/token`. The retired CIRU package
(`/home/dank/ling3-ciru`) documents a source-build fallback for gfx1151.
**ROCm flash-attention requirement:** the vision encoder needs the Triton
flash-attention path. Set `FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE` (in
`frank-default-small.env`) and keep the pure-Python `flash_attn` package
(copied from the CIRU venv) plus a `flash_attn_2_cuda` stub in the venv so
`flash_attn.flash_attn_triton_amd` is findable. Without it the mm encoder
falls back to torch SDPA and engine init OOMs (256 GiB score matrix).
**Tool calling (frank-default-small):** the server runs
`--enable-auto-tool-choice --tool-call-parser qwen3_xml` with a custom
chat template (`/home/dank/models/qwen36-chat_template-tools.jinja`).
The template adds a concrete worked tool-call example and defaults tool
requests to non-thinking (Qwen3 tool calls only work reliably with
thinking off; normal requests keep thinking). The Avesed INT8 quant
emitted malformed Qwen3 XML tool calls; the Quark W8A8 model follows the
format cleanly.
### Env var naming rule — HIP runtime collision ### Env var naming rule — HIP runtime collision
@@ -171,7 +163,7 @@ graph TD
- `sudo` on both boxes needs a tty. Remote sudo steps fail non-interactively. Use `echo '<password>' | sudo -S <command>`. - `sudo` on both boxes needs a tty. Remote sudo steps fail non-interactively. Use `echo '<password>' | sudo -S <command>`.
- Never run `pkill -f` with a pattern that appears in your own command line. It kills your ssh session. Use `pkill -x <exact-name>`. - Never run `pkill -f` with a pattern that appears in your own command line. It kills your ssh session. Use `pkill -x <exact-name>`.
- Model load is slow: Dankserver ~5 minutes cold (15.7 GB IQ4_XS through RAM, `--no-mmap`); Dankclaw vLLM ~5-10 minutes (38 GB checkpoint). Health returns 503 while loading. Do not kill the process. `deploy.sh` waits up to 10 minutes on Dankserver and 20 minutes on Dankclaw (see `HEALTH_WAIT`). - Model load is slow: Dankserver ~5 minutes cold (15.7 GB IQ4_XS through RAM, `--no-mmap`); Dankclaw CIRU ~5-10 minutes (34.9 GiB ActiveFPX payload). Health returns 503 while loading. Do not kill the process. `deploy.sh` waits up to 10 minutes on Dankserver and 20 minutes on Dankclaw (see `HEALTH_WAIT`).
- A client abort and a server release look different in the journal. A client abort logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in [docs/deployment.md](docs/deployment.md). - A client abort and a server release look different in the journal. A client abort logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in [docs/deployment.md](docs/deployment.md).
- Retired hardware history: the Dankserver 35B cuBLAS crash and the DFlash premature-EOS bug are historical; see [docs/dankserver-vram-growth.md](docs/dankserver-vram-growth.md) and [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md). The FastFlowLM NPU endpoint is retired; see [docs/FastFlowLM NPU Inference.md](docs/FastFlowLM%20NPU%20Inference.md). - Retired hardware history: the Dankserver 35B cuBLAS crash and the DFlash premature-EOS bug are historical; see [docs/dankserver-vram-growth.md](docs/dankserver-vram-growth.md) and [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md). The FastFlowLM NPU endpoint is retired; see [docs/FastFlowLM NPU Inference.md](docs/FastFlowLM%20NPU%20Inference.md).
docs/ciru-activefpx-promptforge.md
+167
View File
@@ -0,0 +1,167 @@
---
type: reference
title: CIRU ActiveFPX PromptForge on Dankclaw
description: jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge served by the CIRU ROCmFPX llama.cpp runtime on the Strix Halo iGPU. Install runbook, ROCm 7.15 toolchain, and the frank-default-small service.
tags:
- llama.cpp
- rocm
- dankclaw
- gfx1151
- ciru
- promptforge
timestamp: 2026-08-18
---
# CIRU ActiveFPX PromptForge on Dankclaw
`frank-default-small` (port 52625) now serves
`jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` through the CIRU ROCmFPX
llama.cpp runtime. This replaced the retired vLLM service that previously
occupied the slot.
## What This Is
The model is a **GGUF**, not a vLLM checkpoint. It is the quality-and-speed
release of Qwen3.8-27B for AMD ROCm. CIRU DualView keeps a compact ActiveFPX
GGUF for generation and load optional PromptForge companion views to
accelerate large prompts. The model carries native depth-4 multi-token
prediction (MTP).
| Artifact | Size |
|----------|------|
| `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-v2.gguf` | 15.19 GiB |
| `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-FFN.pfs` | 15.95 GiB |
| `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-GDN.pfs` | 3.75 GiB |
| Three-file payload | ~34.89 GiB |
The two `.pfs` companions are optional. Omit them to run the GGUF alone and
save ~19.7 GiB of memory. The GGUF remains independently usable.
## Requirements
- **ROCm 7.15 is required** for the advertised Strix Halo performance.
The system ROCm (`/opt/rocm-7.2.4`) selects slower or fallback kernels.
- The pinned runtime is `ciru-ai/ROCmFPX` tag
`qwen3.8-activefpx-promptforge-v2.3` (commit `103af8b41`).
- Composable Kernel at `fdf4bb7fcc984811cef48ce817d89aac064b984a`.
- Validated GPU: AMD Radeon 8060S / gfx1151.
## ROCm 7.15 Toolchain
The retired `ling3-ciru` package left a complete TheRock ROCm 7.15 SDK
(`7.15.0a20260728`) inside its venv:
```
/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/
_rocm_sdk_devel/ # hipcc, amdclang 23, cmake, headers
_rocm_sdk_libraries/ # hipblas, hipblaslt, rocroller
_rocm_sdk_core/ # libamdhip64, libhsa-runtime64
rocm-sdk-device-gfx1151
```
The compiler is amdclang 23. `rocm-sdk version` reports `7.15.0a20260728`.
The binary is built against and run with these 7.15 libraries.
## Build Runbook (2026-08-18)
Performed on Dankclaw:
1. Install host build tools (ninja-build was missing): `sudo apt install ninja-build`.
2. Clone the runtime and CK at their pinned revisions:
```
git clone --depth 1 --branch qwen3.8-activefpx-promptforge-v2.3 \
https://github.com/ciru-ai/ROCmFPX.git
git clone --depth 1 https://github.com/ROCm/composable_kernel.git
git -C composable_kernel checkout fdf4bb7fcc984811cef48ce817d89aac064b984a
```
3. CK headers are generated at CK configure time (`ck/config.h`,
`ck/version.h`). The llama.cpp build includes the CK source tree, so run
CK's configure once and copy the headers into `include/ck/`:
```
cmake -S composable_kernel -B build-ck -DAMDGPU_TARGETS=gfx1151 -DCK_ENABLE_WMMA=ON
cp build-ck/include/ck/config.h build-ck/include/ck/version.h composable_kernel/include/ck/
```
4. Configure the llama.cpp build against the venv ROCm 7.15 SDK:
```
export ROCM_PATH=$SDK/_rocm_sdk_devel
cmake -S ROCmFPX -B build-promptforge -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_C_COMPILER=$ROCM_PATH/llvm/bin/clang \
-DCMAKE_CXX_COMPILER=$ROCM_PATH/llvm/bin/clang++ \
-DCMAKE_HIP_COMPILER=$ROCM_PATH/llvm/bin/clang++ \
-DCMAKE_PREFIX_PATH=$ROCM_PATH \
-DGGML_HIP=ON -DGGML_CUDA=OFF -DGGML_VULKAN=OFF \
-DGGML_HIP_FORCE_MMQ=ON -DGGML_HIP_GRAPHS=ON \
-DGGML_HIP_MMQ_MFMA=ON -DGGML_HIP_NO_VMM=ON \
-DGGML_HIP_ROCWMMA_FATTN=OFF -DGGML_NATIVE=ON \
-DAMDGPU_TARGETS=gfx1151 -DGPU_BUILD_TARGETS=gfx1151 \
-DPROMPTFORGE_CK_ROOT=$CK_ROOT \
-DLLAMA_BUILD_SERVER=ON \
-DLLAMA_BUILD_WEBUI=OFF
cmake --build build-promptforge --target llama-server -j$(nproc)
```
`LLAMA_BUILD_WEBUI=OFF` is required. The embedded browser UI downloads
assets from an HF bucket (`llama-ui`) that returns 404; the service is
consumed through the OpenAI-compatible API, so the UI is not needed.
Build output: `build-promptforge/bin/llama-server`.
```mermaid
graph TD
A[Clone ROCmFPX v2.3 + CK pinned] --> B[Configure CK, copy generated headers]
B --> C[Configure llama-server vs ROCm 7.15 SDK]
C --> D[Build llama-server]
D --> E[Verify --list-devices shows gfx1151]
E --> F[Download v2 GGUF + FFN + GDN]
F --> G[Deploy frank-default-small]
G --> H[Health check + chat completion]
```
## Model Download
Downloaded with the HF token to
`/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX/`.
## Service
- Unit: `frank-default-small.service`, port 52625
- Script: `scripts/serve-frank-small.sh`, env `scripts/frank-default-small.env`
- The script exports the ROCm 7.15 SDK libraries in `LD_LIBRARY_PATH` before
the system ROCm. `ROCM_PATH` points at the devel SDK.
### Launch profile (from the model card)
- 64K context, one slot, f16 KV, context checkpoints (32), prompt cache
- Native MTP draft, depth 4, device ROCm0
- `--temp 0.7 --top-p 0.95`, reasoning off
- PromptForge env: `PROMPTFORGE_MODE=m2048_fused_tail1476`,
`GGML_CUDA_GRAPH_OPT=0`, `LLAMA_MTP_CPU_ARGMAX_FASTPATH=1`
### Required environment
```
export PROMPTFORGE_SIDECAR=...FFN.pfs
export PROMPTFORGE_GDN_SIDECAR=...GDN.pfs
export PROMPTFORGE_MODE=m2048_fused_tail1476
export GGML_CUDA_GRAPH_OPT=0
export LLAMA_MTP_CPU_ARGMAX_FASTPATH=1
```
The `p_min=0` MTP profile uses the CPU argmax fast path. This avoids ROCm's
unsupported TOP_K operator.
## Operational Notes
- Health: `curl http://127.0.0.1:52625/health`. Returns 503 while loading.
- Model load is slow (34.9 GiB payload); do not kill the process during
warmup.
- `deploy.sh dankclaw` waits 20 minutes for health.
- The systemd unit must not set `HOST` or `TEMP` env vars (HIP collision
rule). `--host`/`--port`/`--temp` are CLI flags only.
## See Also
- [[frank-redo-2026-08-17]] — the fleet topologies this slot has served.
- [[Shared Memory Architecture Dankclaw]] — unified memory sizing.
- [[qwen3.8-strix-optimizations]] — llama.cpp ROCm0 baselines on this iGPU.
- [[VLLM Overview]] — vLLM reference (retired for this slot).
docs/index.md
+16 -10
View File
@@ -1,8 +1,8 @@
# vLLM Scripts — Knowledge Base # vLLM Scripts — Knowledge Base
Documentation for the LLM inference fleet on two local machines Documentation for the LLM inference fleet on two local machines
(2026-08-17 redo): llama.cpp FastMTP on Dankserver and vLLM ROCm on (2026-08-18): llama.cpp FastMTP on Dankserver and CIRU ROCmFPX llama.cpp
Dankclaw. on Dankclaw.
## Deployment Topology ## Deployment Topology
@@ -19,10 +19,10 @@ FastMTP 32K draft + ngram-mod, 128K ctx]
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04] subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
D[gfx1151 iGPU unified memory] D[gfx1151 iGPU unified memory]
E[vLLM ROCm E[CIRU ROCmFPX llama.cpp
frank-default-small, port 52625] frank-default-small, port 52625]
F[nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8 F[jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge
full 262144 ctx, ngram spec + tools, 4 seqs] 64K ctx, native MTP, PromptForge DualView]
D --> E --> F D --> E --> F
end end
@@ -40,17 +40,23 @@ full 262144 ctx, ngram spec + tools, 4 seqs]
## Active Deployment ## Active Deployment
- [[deployment]] — deploy and update process for both servers - [[deployment]] — deploy and update process for both servers
- [[frank-redo-2026-08-17]] — the full redo: topology change, model - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
choices, context math, and the two install runbooks - [[ciru-activefpx-promptforge]] — the CIRU runtime on Dankclaw
(frank-default-small)
- [[Dankserver]] — the llama.cpp FastMTP box (frank-default-large) - [[Dankserver]] — the llama.cpp FastMTP box (frank-default-large)
- [[Dankclaw]] — the vLLM ROCm box (frank-default-small) - [[Dankclaw]] — the CIRU ROCmFPX box (frank-default-small)
## vLLM Reference (Dankclaw) ## CIRU Reference (Dankclaw)
- [[ciru-activefpx-promptforge]] — model, ROCm 7.15 toolchain, build,
and the frank-default-small service
## vLLM Reference (historical)
- [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism - [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism
- [[VLLM Index]] — quick reference with commands, endpoints, and parameter table - [[VLLM Index]] — quick reference with commands, endpoints, and parameter table
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware - [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware
notes and sizing (the model family served by frank-default-small) notes and sizing (the model family formerly served by frank-default-small)
## Memory Management ## Memory Management
scripts/deploy.sh
+4 -3
View File
@@ -16,9 +16,9 @@
# ./deploy.sh <target> [--dry-run] # ./deploy.sh <target> [--dry-run]
# FRANK_SUDO_PW=... ./deploy.sh <target> # FRANK_SUDO_PW=... ./deploy.sh <target>
# #
# Targets (2026-08-17 topology): # Targets (2026-08-18 topology):
# dankserver - llama.cpp FastMTP (frank-default-large.service, port 7770) # dankserver - llama.cpp FastMTP (frank-default-large.service, port 7770)
# dankclaw - vLLM ROCm (frank-default-small.service, port 52625) # dankclaw - CIRU ROCmFPX llama.cpp (frank-default-small.service, port 52625)
set -euo pipefail set -euo pipefail
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@@ -41,7 +41,8 @@ REMOTE_DIR[dankclaw]="/home/dank/vllmScripts"
SERVICE[dankclaw]="frank-default-small" SERVICE[dankclaw]="frank-default-small"
SERVICE_HEALTH[dankclaw]="http://127.0.0.1:52625/health" SERVICE_HEALTH[dankclaw]="http://127.0.0.1:52625/health"
DROP_CACHES[dankclaw]=0 DROP_CACHES[dankclaw]=0
# vLLM loads a 38 GB checkpoint; allow 20 minutes. # CIRU llama.cpp (ROCm 7.15) loads the 34.9 GiB ActiveFPX payload + MTP
# draft and initializes PromptForge views; allow 20 minutes.
HEALTH_WAIT[dankclaw]=600 HEALTH_WAIT[dankclaw]=600
TARGET="${1:-}" TARGET="${1:-}"
scripts/frank-default-small.env
+54 -61
View File
@@ -1,72 +1,65 @@
# Tunables for frank-default-small (vLLM, dankclaw, Strix Halo gfx1151). # Tunables for frank-default-small (CIRU llama.cpp, dankclaw, Strix Halo
# gfx1151).
# #
# Runtime: vLLM (ROCm) in /home/dank/vllm-venv serving # Runtime: the PromptForge llama-server built from ciru-ai/ROCmFPX tag
# Avesed/Qwen3.6-35B-A3B-INT8-W8A8 (compressed-tensors INT8 W8A8, MoE 35B # qwen3.8-activefpx-promptforge-v2.3 serving
# total / 3B active, Qwen3.5-MoE hybrid: 30 of 40 layers are GatedDeltaNet # jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge (v2). ActiveFPX compact
# linear attention). Alias frank-default-small, port 52625. # GGUF with PromptForge companions for CIRU DualView prompt acceleration and
# native depth-4 MTP. Alias frank-default-small, port 52625.
# #
# The ROCm vLLM install was created with: # This runtime replaced the retired vLLM service that previously occupied
# uv venv --python /usr/bin/python3.12 /home/dank/vllm-venv # this slot. See docs/ciru-activefpx-promptforge.md.
# uv pip install --python /home/dank/vllm-venv vllm --torch-backend=auto #
# The --torch-backend=auto resolver pulls the AMD ROCm wheels (torch + # WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The ROCm
# amd-torch-device-gfx1151 + vllm rocm variant) from the AMD index. # HIP runtime reads those bare env vars (see the 2026-08-13 TEMP incident).
# llama.cpp reads --host/--port/--temp as CLI flags, never as env vars.
# CIRU llama-server binary (ROCmFPX v2.3 build, see docs/ciru runtime page).
# Built against the private TheRock ROCm 7.15 SDK kept in the retired
# ling3-ciru venv. ROCm 7.15 is required for the advertised Strix Halo
# performance; the system ROCm 7.2.4 selects slower/fallback kernels.
LLAMA_BIN=/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server
# Model files. The three-file payload (v2 GGUF + FFN + GDN) enables CIRU
# DualView prompt acceleration; omit the two .pfs files to run the GGUF alone
# and save ~19.7 GiB of memory at the cost of prompt throughput.
MODEL_DIR=/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX
MODEL_FILE=Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-v2.gguf
FFN_PFS=Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-FFN.pfs
GDN_PFS=Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-GDN.pfs
# ROCm 7.15 SDK paths (inside the retired ling3-ciru venv). The serve script
# exports ROCM_PATH and LD_LIBRARY_PATH from these so the 7.15 libraries
# precede any system ROCm 7.2.4.
ROCM_DEVEL=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_devel
ROCM_LIB=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_libraries
ROCM_CORE=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_core
VLLM_BIN=/home/dank/vllm-venv/bin/vllm
# 2026-08-17 (evening): swapped from Avesed/Qwen3.6-35B-A3B-INT8-W8A8 to
# nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8 (AMD Quark W8A8, same
# architecture, vLLM-native expert layout). The Avesed quant's tool-call
# XML fidelity was poor (malformed <function> tags); the Quark model
# follows the Qwen3 XML tool format cleanly.
MODEL=nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8
ALIAS=frank-default-small ALIAS=frank-default-small
BIND_ADDR=0.0.0.0 BIND_ADDR=0.0.0.0
LISTEN_PORT=52625 LISTEN_PORT=52625
# Full native context. The KV cache stays unquantized (auto = model dtype, # Serving defaults from the model card: 64K context, single slot, f16 KV,
# bf16). Worst-case KV at 262144 x 4 sequences is ~10.7 GiB; weights are # context checkpoints, prompt cache. Native MTP depth-4.
# ~35 GiB; total fits the Strix Halo unified pool with room to spare. CTX_SIZE=65536
MAX_MODEL_LEN=262144 PARALLEL=1
KV_CACHE_DTYPE=auto BATCH_SIZE=2048
UBATCH_SIZE=2048
CACHE_TYPE_K=f16
CACHE_TYPE_V=f16
CTX_CHECKPOINTS=32
CACHE_RAM=8192
# Concurrency and batching (user request). max-num-batched-tokens lowered # Native MTP draft (depth-4, greedy, ROCm-safe CPU argmax fast path).
# to 8192 on 2026-08-17 (evening). vLLM has no separate ubatch knob. SPEC_DRAFT_N_MAX=4
MAX_NUM_SEQS=4 SPEC_DRAFT_N_MIN=0
MAX_NUM_BATCHED_TOKENS=8192 SPEC_DRAFT_P_MIN=0.0
SPEC_DRAFT_P_SPLIT=0.10
# vLLM has no separate ubatch knob. The scheduler issues one batch per step # Sampling (model card defaults).
# with up to max-num-batched-tokens tokens; the KV cache is preallocated at TEMPERATURE=0.7
# max-model-len. Set gpu-memory-utilization below 1.0 so the unified pool TOP_P=0.95
# keeps headroom for the host.
GPU_MEMORY_UTILIZATION=0.92
# Prefix caching. This model is hybrid (GatedDeltaNet + full attention); vLLM REASONING=off
# keeps prefix caching OPT-IN by default for hybrid models ("while the feature
# matures"), so it is off unless we ask for it. Turning it on reuses the KV of
# the shared prompt prefix (pi re-sends the same system prompt + tool schema
# every turn), so follow-up requests skip re-prefilling thousands of tokens.
# Big TTFT win for the agent loop. Set 0 to disable.
ENABLE_PREFIX_CACHING=1
# ngram speculative decoding (vLLM ngram proposer). VERBOSITY=4
# prompt_lookup_max = ngram window; num_speculative_tokens = draft depth.
NGRAM_LOOKUP_MAX=5
NGRAM_LOOKUP_MIN=1
NUM_SPEC_TOKENS=5
# Tool calling. Clients (pi, OpenWebUI) send tool_choice=auto; without
# --enable-auto-tool-choice vLLM rejects it with HTTP 400. The Avesed
# chat template renders tools in the Qwen3 XML format, so the qwen3_xml
# parser is the matching one. CHAT_TEMPLATE must point at the repo's
# chat_template.jinja (the tokenizer_config.json ships none; vLLM's
# built-in template does not render the XML tool instructions).
ENABLE_AUTO_TOOL_CHOICE=1
TOOL_CALL_PARSER=qwen3_xml
CHAT_TEMPLATE=/home/dank/models/qwen36-chat_template-tools.jinja
# RDNA flash-attention for the vision encoder (and general FA paths).
# Without this, the mm encoder falls back to torch SDPA and the warmup
# OOMs (256 GiB score matrix at max_model_len; measured 2026-08-17).
# Requires the flash_attn package with flash_attn_triton_amd findable
# (the pure-Python copy from the CIRU package plus a flash_attn_2_cuda
# stub; see docs/frank-redo-2026-08-17.md).
FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE
scripts/frank-default-small.service
+1 -2
View File
@@ -1,5 +1,5 @@
[Unit] [Unit]
Description=vLLM Qwen3.6-35B-A3B INT8 W8A8 ngram spec (dankclaw, frank-default-small) Description=CIRU llama.cpp Qwen3.8-27B ActiveFPX PromptForge ngram MTP (dankclaw, frank-default-small)
After=network-online.target After=network-online.target
Wants=network-online.target Wants=network-online.target
@@ -8,7 +8,6 @@ Type=simple
User=dank User=dank
Group=dank Group=dank
Environment="FRANK_SMALL_ENV_FILE=/home/dank/vllmScripts/scripts/frank-default-small.env" Environment="FRANK_SMALL_ENV_FILE=/home/dank/vllmScripts/scripts/frank-default-small.env"
Environment="HF_HOME=/home/dank/.cache/huggingface"
ExecStart=/home/dank/vllmScripts/scripts/serve-frank-small.sh ExecStart=/home/dank/vllmScripts/scripts/serve-frank-small.sh
Restart=on-failure Restart=on-failure
RestartSec=10 RestartSec=10
scripts/serve-frank-small.sh
+91 -47
View File
@@ -1,10 +1,16 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Serve Avesed/Qwen3.6-35B-A3B-INT8-W8A8 via vLLM on Dankclaw (Strix Halo). # Serve jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge via the CIRU ROCmFPX
# llama.cpp runtime on Dankclaw (Strix Halo gfx1151).
# #
# Runs the ROCm vLLM build in /home/dank/vllm-venv. The model is a # Runtime: the PromptForge llama-server built from ciru-ai/ROCmFPX tag
# compressed-tensors INT8 W8A8 quant (MoE, 35B total / 3B active). Full # qwen3.8-activefpx-promptforge-v2.3 (commit 103af8b41) against the private
# 262144 context, unquantized KV cache, 4 concurrent sequences, ngram # TheRock ROCm 7.15 SDK (kept in the retired ling3-ciru venv). The model is
# speculative decoding. # a GGUF with optional PromptForge companion views (FFN/GDN .pfs) that enable
# CIRU DualView prompt acceleration. Native depth-4 MTP speculative decoding.
#
# The model card requires ROCm 7.15 (not the system ROCm 7.2.4) for the
# advertised Strix Halo performance. LD_LIBRARY_PATH therefore points at the
# 7.15 SDK libraries first. An older system ROCm must not precede them.
# #
# Tunables come from frank-default-small.env in this directory unless # Tunables come from frank-default-small.env in this directory unless
# FRANK_SMALL_ENV_FILE overrides that path. # FRANK_SMALL_ENV_FILE overrides that path.
@@ -20,55 +26,93 @@ if [[ -f "$ENV_FILE" ]]; then
fi fi
# --- defaults (used when the env file does not set them) ------------------- # --- defaults (used when the env file does not set them) -------------------
: "${VLLM_BIN:=/home/dank/vllm-venv/bin/vllm}" : "${LLAMA_BIN:=/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server}"
: "${MODEL:=nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8}" : "${MODEL_DIR:=/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX}"
: "${MODEL_FILE:=Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-v2.gguf}"
: "${FFN_PFS:=Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-FFN.pfs}"
: "${GDN_PFS:=Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-GDN.pfs}"
: "${ROCM_DEVEL:=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_devel}"
: "${ROCM_LIB:=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_libraries}"
: "${ROCM_CORE:=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_core}"
: "${ALIAS:=frank-default-small}" : "${ALIAS:=frank-default-small}"
: "${BIND_ADDR:=0.0.0.0}" : "${BIND_ADDR:=0.0.0.0}"
: "${LISTEN_PORT:=52625}" : "${LISTEN_PORT:=52625}"
: "${MAX_MODEL_LEN:=262144}" : "${CTX_SIZE:=65536}"
: "${KV_CACHE_DTYPE:=auto}" : "${PARALLEL:=1}"
: "${MAX_NUM_SEQS:=4}" : "${BATCH_SIZE:=2048}"
: "${MAX_NUM_BATCHED_TOKENS:=8192}" : "${UBATCH_SIZE:=2048}"
: "${GPU_MEMORY_UTILIZATION:=0.92}" : "${CACHE_TYPE_K:=f16}"
: "${NGRAM_LOOKUP_MAX:=5}" : "${CACHE_TYPE_V:=f16}"
: "${NGRAM_LOOKUP_MIN:=1}" : "${CTX_CHECKPOINTS:=32}"
: "${NUM_SPEC_TOKENS:=5}" : "${CACHE_RAM:=8192}"
: "${ENABLE_AUTO_TOOL_CHOICE:=1}" : "${SPEC_DRAFT_N_MAX:=4}"
: "${TOOL_CALL_PARSER:=qwen3_xml}" : "${SPEC_DRAFT_N_MIN:=0}"
: "${ENABLE_PREFIX_CACHING:=1}" : "${SPEC_DRAFT_P_MIN:=0.0}"
# The Avesed tokenizer_config.json has no chat_template; vLLM would use its : "${SPEC_DRAFT_P_SPLIT:=0.10}"
# built-in template which does not render the model's XML tool format. Use : "${TEMPERATURE:=0.7}"
# the repo's chat_template.jinja (ships the <tool_call> instructions). : "${TOP_P:=0.95}"
: "${CHAT_TEMPLATE:=/home/dank/models/qwen36-chat_template-tools.jinja}" : "${REASONING:=off}"
: "${VERBOSITY:=4}"
if [[ ! -x "$VLLM_BIN" ]]; then MODEL="$MODEL_DIR/$MODEL_FILE"
echo "missing vllm binary: $VLLM_BIN" >&2 export PROMPTFORGE_SIDECAR="$MODEL_DIR/$FFN_PFS"
export PROMPTFORGE_GDN_SIDECAR="$MODEL_DIR/$GDN_PFS"
export PROMPTFORGE_MODE="m2048_fused_tail1476"
export GGML_CUDA_GRAPH_OPT=0
export LLAMA_MTP_CPU_ARGMAX_FASTPATH=1
if [[ ! -f "$MODEL" ]]; then
echo "missing required file: $MODEL" >&2
exit 1
fi
if [[ ! -f "$PROMPTFORGE_SIDECAR" ]]; then
echo "missing PromptForge FFN companion: $PROMPTFORGE_SIDECAR" >&2
exit 1
fi
if [[ ! -f "$PROMPTFORGE_GDN_SIDECAR" ]]; then
echo "missing PromptForge GDN companion: $PROMPTFORGE_GDN_SIDECAR" >&2
exit 1
fi
if [[ ! -x "$LLAMA_BIN" ]]; then
echo "missing llama-server binary: $LLAMA_BIN" >&2
exit 1 exit 1
fi fi
TOOL_ARGS=() # ROCm 7.15 SDK libraries must precede any system ROCm in LD_LIBRARY_PATH.
if [[ "${ENABLE_AUTO_TOOL_CHOICE}" == "1" ]]; then export ROCM_PATH="$ROCM_DEVEL"
TOOL_ARGS+=(--enable-auto-tool-choice) export LD_LIBRARY_PATH="${ROCM_LIB}/lib:${ROCM_CORE}/lib:${ROCM_DEVEL}/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
TOOL_ARGS+=(--tool-call-parser "${TOOL_CALL_PARSER}")
fi
PC_ARGS=() exec "$LLAMA_BIN" \
if [[ "${ENABLE_PREFIX_CACHING}" == "1" ]]; then --model "$MODEL" \
PC_ARGS+=(--enable-prefix-caching) --alias "$ALIAS" \
fi
exec "$VLLM_BIN" serve "$MODEL" \
--host "$BIND_ADDR" \ --host "$BIND_ADDR" \
--port "$LISTEN_PORT" \ --port "$LISTEN_PORT" \
--served-model-name "$ALIAS" \ -dev ROCm0 \
--max-model-len "$MAX_MODEL_LEN" \ -ngl 999 \
--kv-cache-dtype "$KV_CACHE_DTYPE" \ -c "$CTX_SIZE" \
--max-num-seqs "$MAX_NUM_SEQS" \ -b "$BATCH_SIZE" \
--max-num-batched-tokens "$MAX_NUM_BATCHED_TOKENS" \ -ub "$UBATCH_SIZE" \
--gpu-memory-utilization "$GPU_MEMORY_UTILIZATION" \ -fa on \
--speculative-config "{\"method\":\"ngram\",\"prompt_lookup_max\":$NGRAM_LOOKUP_MAX,\"prompt_lookup_min\":$NGRAM_LOOKUP_MIN,\"num_speculative_tokens\":$NUM_SPEC_TOKENS}" \ -ctk "$CACHE_TYPE_K" \
--reasoning-parser qwen3 \ -ctv "$CACHE_TYPE_V" \
--trust-remote-code \ -np "$PARALLEL" \
--chat-template "$CHAT_TEMPLATE" \ -ctxcp "$CTX_CHECKPOINTS" \
"${PC_ARGS[@]}" \ --cache-ram "$CACHE_RAM" \
"${TOOL_ARGS[@]}" --cache-prompt \
--metrics \
--spec-type draft-mtp \
--spec-draft-device ROCm0 \
--spec-draft-ngl 999 \
--spec-draft-type-k f16 \
--spec-draft-type-v f16 \
--spec-draft-n-max "$SPEC_DRAFT_N_MAX" \
--spec-draft-n-min "$SPEC_DRAFT_N_MIN" \
--spec-draft-p-min "$SPEC_DRAFT_P_MIN" \
--spec-draft-p-split "$SPEC_DRAFT_P_SPLIT" \
--spec-draft-backend-sampling \
--temp "$TEMPERATURE" \
--top-p "$TOP_P" \
--reasoning "$REASONING" \
--reasoning-format none \
--reasoning-budget -1 \
-lv "$VERBOSITY"