Frank
8e4d63b68d
User request 2026-08-19: lower per-stream context from 262144 to 200000. CTX_SIZE is the total; llama.cpp splits it across the 2 parallel slots. f16 KV drops from ~32 GiB to ~24 GiB, adding OOM headroom. Docs: sync the 2026-08-18 service swap (frank-default-large on dankclaw, frank-default-small on dankserver) across AGENTS.md, README.md, deployment.md, ciru-activefpx-promptforge.md, and index.md. The pi models.json contextWindow/maxTokens for frank-default-large move 262144 -> 200000 (separate file, outside the repo).
171 lines
6.4 KiB
Markdown
171 lines
6.4 KiB
Markdown
---
|
|
type: reference
|
|
title: CIRU ActiveFPX PromptForge on Dankclaw
|
|
description: jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge served by the CIRU ROCmFPX llama.cpp runtime on the Strix Halo iGPU. Install runbook, ROCm 7.15 toolchain, and the frank-default-large service.
|
|
tags:
|
|
- llama.cpp
|
|
- rocm
|
|
- dankclaw
|
|
- gfx1151
|
|
- ciru
|
|
- promptforge
|
|
timestamp: 2026-08-19
|
|
---
|
|
|
|
# CIRU ActiveFPX PromptForge on Dankclaw
|
|
|
|
`frank-default-large` (port 52625) serves
|
|
`jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` through the CIRU ROCmFPX
|
|
llama.cpp runtime. This replaced the retired vLLM service that previously
|
|
occupied the slot.
|
|
|
|
## What This Is
|
|
|
|
The model is a **GGUF**, not a vLLM checkpoint. It is the quality-and-speed
|
|
release of Qwen3.8-27B for AMD ROCm. CIRU DualView keeps a compact ActiveFPX
|
|
GGUF for generation and load optional PromptForge companion views to
|
|
accelerate large prompts. The model carries native depth-4 multi-token
|
|
prediction (MTP).
|
|
|
|
| Artifact | Size |
|
|
|----------|------|
|
|
| `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-v2.gguf` | 15.19 GiB |
|
|
| `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-FFN.pfs` | 15.95 GiB |
|
|
| `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-GDN.pfs` | 3.75 GiB |
|
|
| Three-file payload | ~34.89 GiB |
|
|
|
|
The two `.pfs` companions are optional. Omit them to run the GGUF alone and
|
|
save ~19.7 GiB of memory. The GGUF remains independently usable.
|
|
|
|
## Requirements
|
|
|
|
- **ROCm 7.15 is required** for the advertised Strix Halo performance.
|
|
The system ROCm (`/opt/rocm-7.2.4`) selects slower or fallback kernels.
|
|
- The pinned runtime is `ciru-ai/ROCmFPX` tag
|
|
`qwen3.8-activefpx-promptforge-v2.3` (commit `103af8b41`).
|
|
- Composable Kernel at `fdf4bb7fcc984811cef48ce817d89aac064b984a`.
|
|
- Validated GPU: AMD Radeon 8060S / gfx1151.
|
|
|
|
## ROCm 7.15 Toolchain
|
|
|
|
The retired `ling3-ciru` package left a complete TheRock ROCm 7.15 SDK
|
|
(`7.15.0a20260728`) inside its venv:
|
|
|
|
```
|
|
/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/
|
|
_rocm_sdk_devel/ # hipcc, amdclang 23, cmake, headers
|
|
_rocm_sdk_libraries/ # hipblas, hipblaslt, rocroller
|
|
_rocm_sdk_core/ # libamdhip64, libhsa-runtime64
|
|
rocm-sdk-device-gfx1151
|
|
```
|
|
|
|
The compiler is amdclang 23. `rocm-sdk version` reports `7.15.0a20260728`.
|
|
The binary is built against and run with these 7.15 libraries.
|
|
|
|
## Build Runbook (2026-08-18)
|
|
|
|
Performed on Dankclaw:
|
|
|
|
1. Install host build tools (ninja-build was missing): `sudo apt install ninja-build`.
|
|
2. Clone the runtime and CK at their pinned revisions:
|
|
```
|
|
git clone --depth 1 --branch qwen3.8-activefpx-promptforge-v2.3 \
|
|
https://github.com/ciru-ai/ROCmFPX.git
|
|
git clone --depth 1 https://github.com/ROCm/composable_kernel.git
|
|
git -C composable_kernel checkout fdf4bb7fcc984811cef48ce817d89aac064b984a
|
|
```
|
|
3. CK headers are generated at CK configure time (`ck/config.h`,
|
|
`ck/version.h`). The llama.cpp build includes the CK source tree, so run
|
|
CK's configure once and copy the headers into `include/ck/`:
|
|
```
|
|
cmake -S composable_kernel -B build-ck -DAMDGPU_TARGETS=gfx1151 -DCK_ENABLE_WMMA=ON
|
|
cp build-ck/include/ck/config.h build-ck/include/ck/version.h composable_kernel/include/ck/
|
|
```
|
|
4. Configure the llama.cpp build against the venv ROCm 7.15 SDK:
|
|
```
|
|
export ROCM_PATH=$SDK/_rocm_sdk_devel
|
|
cmake -S ROCmFPX -B build-promptforge -G Ninja \
|
|
-DCMAKE_BUILD_TYPE=Release \
|
|
-DCMAKE_C_COMPILER=$ROCM_PATH/llvm/bin/clang \
|
|
-DCMAKE_CXX_COMPILER=$ROCM_PATH/llvm/bin/clang++ \
|
|
-DCMAKE_HIP_COMPILER=$ROCM_PATH/llvm/bin/clang++ \
|
|
-DCMAKE_PREFIX_PATH=$ROCM_PATH \
|
|
-DGGML_HIP=ON -DGGML_CUDA=OFF -DGGML_VULKAN=OFF \
|
|
-DGGML_HIP_FORCE_MMQ=ON -DGGML_HIP_GRAPHS=ON \
|
|
-DGGML_HIP_MMQ_MFMA=ON -DGGML_HIP_NO_VMM=ON \
|
|
-DGGML_HIP_ROCWMMA_FATTN=OFF -DGGML_NATIVE=ON \
|
|
-DAMDGPU_TARGETS=gfx1151 -DGPU_BUILD_TARGETS=gfx1151 \
|
|
-DPROMPTFORGE_CK_ROOT=$CK_ROOT \
|
|
-DLLAMA_BUILD_SERVER=ON \
|
|
-DLLAMA_BUILD_WEBUI=OFF
|
|
cmake --build build-promptforge --target llama-server -j$(nproc)
|
|
```
|
|
`LLAMA_BUILD_WEBUI=OFF` is required. The embedded browser UI downloads
|
|
assets from an HF bucket (`llama-ui`) that returns 404; the service is
|
|
consumed through the OpenAI-compatible API, so the UI is not needed.
|
|
|
|
Build output: `build-promptforge/bin/llama-server`.
|
|
|
|
```mermaid
|
|
graph TD
|
|
A[Clone ROCmFPX v2.3 + CK pinned] --> B[Configure CK, copy generated headers]
|
|
B --> C[Configure llama-server vs ROCm 7.15 SDK]
|
|
C --> D[Build llama-server]
|
|
D --> E[Verify --list-devices shows gfx1151]
|
|
E --> F[Download v2 GGUF + FFN + GDN]
|
|
F --> G[Deploy frank-default-small]
|
|
G --> H[Health check + chat completion]
|
|
```
|
|
|
|
## Model Download
|
|
|
|
Downloaded with the HF token to
|
|
`/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX/`.
|
|
|
|
## Service
|
|
|
|
- Unit: `frank-default-large.service`, port 52625
|
|
- Script: `scripts/serve-frank-large.sh`, env `scripts/frank-default-large.env`
|
|
- The script exports the ROCm 7.15 SDK libraries in `LD_LIBRARY_PATH` before
|
|
the system ROCm. `ROCM_PATH` points at the devel SDK.
|
|
|
|
### Launch profile
|
|
|
|
- 200K context per slot (user request 2026-08-19; native max 262144),
|
|
2 slots, f16 KV (~24 GiB total for both slots), context checkpoints
|
|
(16), prompt cache
|
|
- Native MTP draft, depth 4, device ROCm0
|
|
- `--temp 0.7 --top-p 0.95`, reasoning on (`--reasoning-format deepseek`
|
|
puts `<think>` reasoning in `message.reasoning_content`)
|
|
- PromptForge env: `PROMPTFORGE_MODE=m2048_fused_tail1476`,
|
|
`GGML_CUDA_GRAPH_OPT=0`, `LLAMA_MTP_CPU_ARGMAX_FASTPATH=1`
|
|
|
|
### Required environment
|
|
|
|
```
|
|
export PROMPTFORGE_SIDECAR=...FFN.pfs
|
|
export PROMPTFORGE_GDN_SIDECAR=...GDN.pfs
|
|
export PROMPTFORGE_MODE=m2048_fused_tail1476
|
|
export GGML_CUDA_GRAPH_OPT=0
|
|
export LLAMA_MTP_CPU_ARGMAX_FASTPATH=1
|
|
```
|
|
|
|
The `p_min=0` MTP profile uses the CPU argmax fast path. This avoids ROCm's
|
|
unsupported TOP_K operator.
|
|
|
|
## Operational Notes
|
|
|
|
- Health: `curl http://127.0.0.1:52625/health`. Returns 503 while loading.
|
|
- Model load is slow (34.9 GiB payload); do not kill the process during
|
|
warmup.
|
|
- `deploy.sh dankclaw` waits 20 minutes for health.
|
|
- The systemd unit must not set `HOST` or `TEMP` env vars (HIP collision
|
|
rule). `--host`/`--port`/`--temp` are CLI flags only.
|
|
|
|
## See Also
|
|
|
|
- [[frank-redo-2026-08-17]] — the fleet topologies this slot has served.
|
|
- [[Shared Memory Architecture Dankclaw]] — unified memory sizing.
|
|
- [[qwen3.8-strix-optimizations]] — llama.cpp ROCm0 baselines on this iGPU.
|
|
- [[VLLM Overview]] — vLLM reference (retired for this slot).
|