dankclaw: replace vLLM with CIRU ROCmFPX llama.cpp runtime

Retire the vLLM service (frank-default-small, port 52625) on Dankclaw and serve jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge through the CIRU ROCmFPX llama.cpp runtime instead. - Build llama-server from ciru-ai/ROCmFPX tag qwen3.8-activefpx-promptforge-v2.3 (commit 103af8b41) against the private TheRock ROCm 7.15 SDK in the retired ling3-ciru venv. The model card requires ROCm 7.15 for the advertised Strix Halo performance; the system ROCm 7.2.4 selects slower kernels. CK pinned at fdf4bb7fcc984811cef48ce817d89aac064b984a; generated ck/config.h + ck/version.h copied into the CK source tree so the ggml-hip build can include them. LLAMA_BUILD_WEBUI=OFF (HF bucket llama-ui returns 404; server is consumed via the OpenAI API). - New scripts/serve-frank-small.sh + frank-default-small.env: 64K ctx, one slot, f16 KV, context checkpoints, prompt cache, native depth-4 MTP on ROCm0, PromptForge DualView sidecars (FFN + GDN), temp 0.7 top-p 0.95, reasoning off. LD_LIBRARY_PATH puts the 7.15 SDK libs before any system ROCm. - Update the systemd unit description (same service name and port). - deploy.sh: update the dankclaw target comment. - Docs: new docs/ciru-activefpx-promptforge.md with the build runbook; update docs/index.md and AGENTS.md topology. - The vLLM venvs (vllm-venv, vllm-vk-venv) were moved to /home/dank/.AGENTTRASH/ on the target; the vllm-vulkan plugin and docs were removed from the repo in an earlier commit.
2026-08-17 21:12:35 -07:00
parent 4941d911f2
commit 3b51b5a94a
7 changed files with 359 additions and 157 deletions
@@ -0,0 +1,167 @@
+---
+type: reference
+title: CIRU ActiveFPX PromptForge on Dankclaw
+description: jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge served by the CIRU ROCmFPX llama.cpp runtime on the Strix Halo iGPU. Install runbook, ROCm 7.15 toolchain, and the frank-default-small service.
+tags:
+  - llama.cpp
+  - rocm
+  - dankclaw
+  - gfx1151
+  - ciru
+  - promptforge
+timestamp: 2026-08-18
+---
+
+# CIRU ActiveFPX PromptForge on Dankclaw
+
+`frank-default-small` (port 52625) now serves
+`jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` through the CIRU ROCmFPX
+llama.cpp runtime. This replaced the retired vLLM service that previously
+occupied the slot.
+
+## What This Is
+
+The model is a **GGUF**, not a vLLM checkpoint. It is the quality-and-speed
+release of Qwen3.8-27B for AMD ROCm. CIRU DualView keeps a compact ActiveFPX
+GGUF for generation and load optional PromptForge companion views to
+accelerate large prompts. The model carries native depth-4 multi-token
+prediction (MTP).
+
+| Artifact | Size |
+|----------|------|
+| `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-v2.gguf` | 15.19 GiB |
+| `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-FFN.pfs` | 15.95 GiB |
+| `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-GDN.pfs` | 3.75 GiB |
+| Three-file payload | ~34.89 GiB |
+
+The two `.pfs` companions are optional. Omit them to run the GGUF alone and
+save ~19.7 GiB of memory. The GGUF remains independently usable.
+
+## Requirements
+
+- **ROCm 7.15 is required** for the advertised Strix Halo performance.
+  The system ROCm (`/opt/rocm-7.2.4`) selects slower or fallback kernels.
+- The pinned runtime is `ciru-ai/ROCmFPX` tag
+  `qwen3.8-activefpx-promptforge-v2.3` (commit `103af8b41`).
+- Composable Kernel at `fdf4bb7fcc984811cef48ce817d89aac064b984a`.
+- Validated GPU: AMD Radeon 8060S / gfx1151.
+
+## ROCm 7.15 Toolchain
+
+The retired `ling3-ciru` package left a complete TheRock ROCm 7.15 SDK
+(`7.15.0a20260728`) inside its venv:
+
+```
+/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/
+  _rocm_sdk_devel/      # hipcc, amdclang 23, cmake, headers
+  _rocm_sdk_libraries/  # hipblas, hipblaslt, rocroller
+  _rocm_sdk_core/       # libamdhip64, libhsa-runtime64
+  rocm-sdk-device-gfx1151
+```
+
+The compiler is amdclang 23. `rocm-sdk version` reports `7.15.0a20260728`.
+The binary is built against and run with these 7.15 libraries.
+
+## Build Runbook (2026-08-18)
+
+Performed on Dankclaw:
+
+1. Install host build tools (ninja-build was missing): `sudo apt install ninja-build`.
+2. Clone the runtime and CK at their pinned revisions:
+   ```
+   git clone --depth 1 --branch qwen3.8-activefpx-promptforge-v2.3 \
+     https://github.com/ciru-ai/ROCmFPX.git
+   git clone --depth 1 https://github.com/ROCm/composable_kernel.git
+   git -C composable_kernel checkout fdf4bb7fcc984811cef48ce817d89aac064b984a
+   ```
+3. CK headers are generated at CK configure time (`ck/config.h`,
+   `ck/version.h`). The llama.cpp build includes the CK source tree, so run
+   CK's configure once and copy the headers into `include/ck/`:
+   ```
+   cmake -S composable_kernel -B build-ck -DAMDGPU_TARGETS=gfx1151 -DCK_ENABLE_WMMA=ON
+   cp build-ck/include/ck/config.h build-ck/include/ck/version.h composable_kernel/include/ck/
+   ```
+4. Configure the llama.cpp build against the venv ROCm 7.15 SDK:
+   ```
+   export ROCM_PATH=$SDK/_rocm_sdk_devel
+   cmake -S ROCmFPX -B build-promptforge -G Ninja \
+     -DCMAKE_BUILD_TYPE=Release \
+     -DCMAKE_C_COMPILER=$ROCM_PATH/llvm/bin/clang \
+     -DCMAKE_CXX_COMPILER=$ROCM_PATH/llvm/bin/clang++ \
+     -DCMAKE_HIP_COMPILER=$ROCM_PATH/llvm/bin/clang++ \
+     -DCMAKE_PREFIX_PATH=$ROCM_PATH \
+     -DGGML_HIP=ON -DGGML_CUDA=OFF -DGGML_VULKAN=OFF \
+     -DGGML_HIP_FORCE_MMQ=ON -DGGML_HIP_GRAPHS=ON \
+     -DGGML_HIP_MMQ_MFMA=ON -DGGML_HIP_NO_VMM=ON \
+     -DGGML_HIP_ROCWMMA_FATTN=OFF -DGGML_NATIVE=ON \
+     -DAMDGPU_TARGETS=gfx1151 -DGPU_BUILD_TARGETS=gfx1151 \
+     -DPROMPTFORGE_CK_ROOT=$CK_ROOT \
+     -DLLAMA_BUILD_SERVER=ON \
+     -DLLAMA_BUILD_WEBUI=OFF
+   cmake --build build-promptforge --target llama-server -j$(nproc)
+   ```
+   `LLAMA_BUILD_WEBUI=OFF` is required. The embedded browser UI downloads
+   assets from an HF bucket (`llama-ui`) that returns 404; the service is
+   consumed through the OpenAI-compatible API, so the UI is not needed.
+
+Build output: `build-promptforge/bin/llama-server`.
+
+```mermaid
+graph TD
+    A[Clone ROCmFPX v2.3 + CK pinned] --> B[Configure CK, copy generated headers]
+    B --> C[Configure llama-server vs ROCm 7.15 SDK]
+    C --> D[Build llama-server]
+    D --> E[Verify --list-devices shows gfx1151]
+    E --> F[Download v2 GGUF + FFN + GDN]
+    F --> G[Deploy frank-default-small]
+    G --> H[Health check + chat completion]
+```
+
+## Model Download
+
+Downloaded with the HF token to
+`/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX/`.
+
+## Service
+
+- Unit: `frank-default-small.service`, port 52625
+- Script: `scripts/serve-frank-small.sh`, env `scripts/frank-default-small.env`
+- The script exports the ROCm 7.15 SDK libraries in `LD_LIBRARY_PATH` before
+  the system ROCm. `ROCM_PATH` points at the devel SDK.
+
+### Launch profile (from the model card)
+
+- 64K context, one slot, f16 KV, context checkpoints (32), prompt cache
+- Native MTP draft, depth 4, device ROCm0
+- `--temp 0.7 --top-p 0.95`, reasoning off
+- PromptForge env: `PROMPTFORGE_MODE=m2048_fused_tail1476`,
+  `GGML_CUDA_GRAPH_OPT=0`, `LLAMA_MTP_CPU_ARGMAX_FASTPATH=1`
+
+### Required environment
+
+```
+export PROMPTFORGE_SIDECAR=...FFN.pfs
+export PROMPTFORGE_GDN_SIDECAR=...GDN.pfs
+export PROMPTFORGE_MODE=m2048_fused_tail1476
+export GGML_CUDA_GRAPH_OPT=0
+export LLAMA_MTP_CPU_ARGMAX_FASTPATH=1
+```
+
+The `p_min=0` MTP profile uses the CPU argmax fast path. This avoids ROCm's
+unsupported TOP_K operator.
+
+## Operational Notes
+
+- Health: `curl http://127.0.0.1:52625/health`. Returns 503 while loading.
+- Model load is slow (34.9 GiB payload); do not kill the process during
+  warmup.
+- `deploy.sh dankclaw` waits 20 minutes for health.
+- The systemd unit must not set `HOST` or `TEMP` env vars (HIP collision
+  rule). `--host`/`--port`/`--temp` are CLI flags only.
+
+## See Also
+
+- [[frank-redo-2026-08-17]] — the fleet topologies this slot has served.
+- [[Shared Memory Architecture Dankclaw]] — unified memory sizing.
+- [[qwen3.8-strix-optimizations]] — llama.cpp ROCm0 baselines on this iGPU.
+- [[VLLM Overview]] — vLLM reference (retired for this slot).
@@ -1,8 +1,8 @@
 # vLLM Scripts — Knowledge Base

 Documentation for the LLM inference fleet on two local machines
-(2026-08-17 redo): llama.cpp FastMTP on Dankserver and vLLM ROCm on
-Dankclaw.
+(2026-08-18): llama.cpp FastMTP on Dankserver and CIRU ROCmFPX llama.cpp
+on Dankclaw.

 ## Deployment Topology

@@ -19,10 +19,10 @@ FastMTP 32K draft + ngram-mod, 128K ctx]

    subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
        D[gfx1151 iGPU unified memory]
-        E[vLLM ROCm
+        E[CIRU ROCmFPX llama.cpp
 frank-default-small, port 52625]
-        F[nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8
-full 262144 ctx, ngram spec + tools, 4 seqs]
+        F[jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge
+64K ctx, native MTP, PromptForge DualView]
        D --> E --> F
    end

@@ -40,17 +40,23 @@ full 262144 ctx, ngram spec + tools, 4 seqs]
 ## Active Deployment

 - [[deployment]] — deploy and update process for both servers
- [[frank-redo-2026-08-17]] — the full redo: topology change, model
-  choices, context math, and the two install runbooks
+- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
+- [[ciru-activefpx-promptforge]] — the CIRU runtime on Dankclaw
+  (frank-default-small)
 - [[Dankserver]] — the llama.cpp FastMTP box (frank-default-large)
- [[Dankclaw]] — the vLLM ROCm box (frank-default-small)
+- [[Dankclaw]] — the CIRU ROCmFPX box (frank-default-small)

-## vLLM Reference (Dankclaw)
+## CIRU Reference (Dankclaw)
+
+- [[ciru-activefpx-promptforge]] — model, ROCm 7.15 toolchain, build,
+  and the frank-default-small service
+
+## vLLM Reference (historical)

 - [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism
 - [[VLLM Index]] — quick reference with commands, endpoints, and parameter table
 - [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware
-  notes and sizing (the model family served by frank-default-small)
+  notes and sizing (the model family formerly served by frank-default-small)

 ## Memory Management