dankclaw: replace vLLM with CIRU ROCmFPX llama.cpp runtime

Retire the vLLM service (frank-default-small, port 52625) on Dankclaw
and serve jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge through the CIRU
ROCmFPX llama.cpp runtime instead.

- Build llama-server from ciru-ai/ROCmFPX tag
  qwen3.8-activefpx-promptforge-v2.3 (commit 103af8b41) against the
  private TheRock ROCm 7.15 SDK in the retired ling3-ciru venv. The
  model card requires ROCm 7.15 for the advertised Strix Halo
  performance; the system ROCm 7.2.4 selects slower kernels.
  CK pinned at fdf4bb7fcc984811cef48ce817d89aac064b984a; generated
  ck/config.h + ck/version.h copied into the CK source tree so the
  ggml-hip build can include them. LLAMA_BUILD_WEBUI=OFF (HF bucket
  llama-ui returns 404; server is consumed via the OpenAI API).
- New scripts/serve-frank-small.sh + frank-default-small.env: 64K ctx,
  one slot, f16 KV, context checkpoints, prompt cache, native depth-4
  MTP on ROCm0, PromptForge DualView sidecars (FFN + GDN), temp 0.7
  top-p 0.95, reasoning off. LD_LIBRARY_PATH puts the 7.15 SDK libs
  before any system ROCm.
- Update the systemd unit description (same service name and port).
- deploy.sh: update the dankclaw target comment.
- Docs: new docs/ciru-activefpx-promptforge.md with the build runbook;
  update docs/index.md and AGENTS.md topology.
- The vLLM venvs (vllm-venv, vllm-vk-venv) were moved to
  /home/dank/.AGENTTRASH/ on the target; the vllm-vulkan plugin and
  docs were removed from the repo in an earlier commit.

AGENTS.md

@@ -3,16 +3,18 @@
 ## Overview
 
 This project manages deployment scripts for the LLM inference fleet on two
-local machines. Topology (2026-08-17 redo):
+local machines. Topology (2026-08-18):
 
 | Target | Host | Runtime | Service | Model | Port |
 |--------|------|---------|---------|-------|------|
 | Dankserver | dank@dankserver | llama.cpp (patched FastMTP build) | frank-default-large | HauhauCS Qwen3.8-27B-Uncensored-Aggressive IQ4_XS + FastMTP 32K draft + ngram-mod | 7770 |
-| Dankclaw | dank@dankclaw | vLLM (ROCm, /home/dank/vllm-venv) | frank-default-small | nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8, ngram spec, tool calling | 52625 |
+| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (ROCm 7.15 SDK in the ling3-ciru venv) | frank-default-small | jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP, PromptForge DualView | 52625 |
 
 The 2026-08-17 redo retired the pipeline-parallel split and the FastFlowLM
-NPU endpoint. The RTX 3090 now runs the whole 27B model standalone
+NPU endpoint. The RTX 3090 runs the whole 27B model standalone
-(no RPC worker). Dankclaw runs vLLM on the Strix Halo iGPU (gfx1151).
+(no RPC worker). The 2026-08-18 swap retired the vLLM service on Dankclaw;
+frank-default-small now runs the CIRU ROCmFPX llama.cpp runtime on the
+Strix Halo iGPU (gfx1151). See docs/ciru-activefpx-promptforge.md.
 The old scripts live in `old/scripts/` for reference; they describe the
 retired PP topology.
 
@@ -60,7 +62,7 @@ Use `scripts/deploy.sh` to deploy committed changes to a target. It pushes `main
 
 ```
 FRANK_SUDO_PW=... ./scripts/deploy.sh dankserver   # llama.cpp FastMTP, port 7770
-FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw     # vLLM, port 52625
+FRANK_SUDO_PW=... ./scripts/deploy.sh dankclaw     # CIRU llama.cpp, port 52625
 ./scripts/deploy.sh dankserver --dry-run           # preview without changes
 ```
 
@@ -96,37 +98,27 @@ back to a slow path and prefill collapses ~12x (measured 2026-08-17: 95 t/s
 vs 1300+ t/s at an 8K prompt on the 3090). See the HauhauCS HF repo README
 for the base recipe.
 
-### vLLM ROCm install (Dankclaw)
+### CIRU ROCmFPX llama.cpp (Dankclaw)
 
-vLLM runs from `/home/dank/vllm-venv` (Python 3.12). Install recipe:
+The CIRU llama-server serves `jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge`
+from `/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server`.
+The runtime is the ciru-ai/ROCmFPX tag `qwen3.8-activefpx-promptforge-v2.3`
+(commit `103af8b41`) built against the private TheRock ROCm 7.15 SDK kept
+in the retired ling3-ciru venv
+(`.venv/lib/python3.12/site-packages/_rocm_sdk_*`). The model card requires
+ROCm 7.15 for the advertised Strix Halo performance; the system ROCm 7.2.4
+selects slower kernels. The serve script exports the 7.15 SDK libraries
+first in `LD_LIBRARY_PATH`.
 
-```
+The model files (v2 GGUF + FFN.pfs + GDN.pfs, ~34.9 GiB) live in
-uv venv --python /usr/bin/python3.12 /home/dank/vllm-venv
+`/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX/`. The full build runbook is
-uv pip install --python /home/dank/vllm-venv vllm --torch-backend=auto
+in docs/ciru-activefpx-promptforge.md. The retired vLLM venvs
-```
+(`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to
+`/home/dank/.AGENTTRASH/` on 2026-08-18.
 
-`--torch-backend=auto` resolves the AMD ROCm wheel set (torch +
+**Env var naming rule still applies:** the HIP runtime reads `TEMP`, `HOST`,
-amd-torch-device-gfx1151 + the vllm rocm variant) from the AMD index.
+and `GPU_MAX_HEAP_SIZE`. The serve script passes `--host`/`--port`/`--temp`
-The model cache is the default HF cache
+as CLI flags only.
-(`/home/dank/.cache/huggingface`); the HF token lives at
-`~/.cache/huggingface/token`. The retired CIRU package
-(`/home/dank/ling3-ciru`) documents a source-build fallback for gfx1151.
-
-**ROCm flash-attention requirement:** the vision encoder needs the Triton
-flash-attention path. Set `FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE` (in
-`frank-default-small.env`) and keep the pure-Python `flash_attn` package
-(copied from the CIRU venv) plus a `flash_attn_2_cuda` stub in the venv so
-`flash_attn.flash_attn_triton_amd` is findable. Without it the mm encoder
-falls back to torch SDPA and engine init OOMs (256 GiB score matrix).
-
-**Tool calling (frank-default-small):** the server runs
-`--enable-auto-tool-choice --tool-call-parser qwen3_xml` with a custom
-chat template (`/home/dank/models/qwen36-chat_template-tools.jinja`).
-The template adds a concrete worked tool-call example and defaults tool
-requests to non-thinking (Qwen3 tool calls only work reliably with
-thinking off; normal requests keep thinking). The Avesed INT8 quant
-emitted malformed Qwen3 XML tool calls; the Quark W8A8 model follows the
-format cleanly.
 
 ### Env var naming rule — HIP runtime collision
 
@@ -171,7 +163,7 @@ graph TD
 
 - `sudo` on both boxes needs a tty. Remote sudo steps fail non-interactively. Use `echo '<password>' | sudo -S <command>`.
 - Never run `pkill -f` with a pattern that appears in your own command line. It kills your ssh session. Use `pkill -x <exact-name>`.
-- Model load is slow: Dankserver ~5 minutes cold (15.7 GB IQ4_XS through RAM, `--no-mmap`); Dankclaw vLLM ~5-10 minutes (38 GB checkpoint). Health returns 503 while loading. Do not kill the process. `deploy.sh` waits up to 10 minutes on Dankserver and 20 minutes on Dankclaw (see `HEALTH_WAIT`).
+- Model load is slow: Dankserver ~5 minutes cold (15.7 GB IQ4_XS through RAM, `--no-mmap`); Dankclaw CIRU ~5-10 minutes (34.9 GiB ActiveFPX payload). Health returns 503 while loading. Do not kill the process. `deploy.sh` waits up to 10 minutes on Dankserver and 20 minutes on Dankclaw (see `HEALTH_WAIT`).
 - A client abort and a server release look different in the journal. A client abort logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in [docs/deployment.md](docs/deployment.md).
 - Retired hardware history: the Dankserver 35B cuBLAS crash and the DFlash premature-EOS bug are historical; see [docs/dankserver-vram-growth.md](docs/dankserver-vram-growth.md) and [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md). The FastFlowLM NPU endpoint is retired; see [docs/FastFlowLM NPU Inference.md](docs/FastFlowLM%20NPU%20Inference.md).
 

docs/ciru-activefpx-promptforge.md

@@ -0,0 +1,167 @@
+---
+type: reference
+title: CIRU ActiveFPX PromptForge on Dankclaw
+description: jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge served by the CIRU ROCmFPX llama.cpp runtime on the Strix Halo iGPU. Install runbook, ROCm 7.15 toolchain, and the frank-default-small service.
+tags:
+  - llama.cpp
+  - rocm
+  - dankclaw
+  - gfx1151
+  - ciru
+  - promptforge
+timestamp: 2026-08-18
+---
+
+# CIRU ActiveFPX PromptForge on Dankclaw
+
+`frank-default-small` (port 52625) now serves
+`jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` through the CIRU ROCmFPX
+llama.cpp runtime. This replaced the retired vLLM service that previously
+occupied the slot.
+
+## What This Is
+
+The model is a **GGUF**, not a vLLM checkpoint. It is the quality-and-speed
+release of Qwen3.8-27B for AMD ROCm. CIRU DualView keeps a compact ActiveFPX
+GGUF for generation and load optional PromptForge companion views to
+accelerate large prompts. The model carries native depth-4 multi-token
+prediction (MTP).
+
+| Artifact | Size |
+|----------|------|
+| `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-v2.gguf` | 15.19 GiB |
+| `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-FFN.pfs` | 15.95 GiB |
+| `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-GDN.pfs` | 3.75 GiB |
+| Three-file payload | ~34.89 GiB |
+
+The two `.pfs` companions are optional. Omit them to run the GGUF alone and
+save ~19.7 GiB of memory. The GGUF remains independently usable.
+
+## Requirements
+
+- **ROCm 7.15 is required** for the advertised Strix Halo performance.
+  The system ROCm (`/opt/rocm-7.2.4`) selects slower or fallback kernels.
+- The pinned runtime is `ciru-ai/ROCmFPX` tag
+  `qwen3.8-activefpx-promptforge-v2.3` (commit `103af8b41`).
+- Composable Kernel at `fdf4bb7fcc984811cef48ce817d89aac064b984a`.
+- Validated GPU: AMD Radeon 8060S / gfx1151.
+
+## ROCm 7.15 Toolchain
+
+The retired `ling3-ciru` package left a complete TheRock ROCm 7.15 SDK
+(`7.15.0a20260728`) inside its venv:
+
+```
+/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/
+  _rocm_sdk_devel/      # hipcc, amdclang 23, cmake, headers
+  _rocm_sdk_libraries/  # hipblas, hipblaslt, rocroller
+  _rocm_sdk_core/       # libamdhip64, libhsa-runtime64
+  rocm-sdk-device-gfx1151
+```
+
+The compiler is amdclang 23. `rocm-sdk version` reports `7.15.0a20260728`.
+The binary is built against and run with these 7.15 libraries.
+
+## Build Runbook (2026-08-18)
+
+Performed on Dankclaw:
+
+1. Install host build tools (ninja-build was missing): `sudo apt install ninja-build`.
+2. Clone the runtime and CK at their pinned revisions:
+   ```
+   git clone --depth 1 --branch qwen3.8-activefpx-promptforge-v2.3 \
+     https://github.com/ciru-ai/ROCmFPX.git
+   git clone --depth 1 https://github.com/ROCm/composable_kernel.git
+   git -C composable_kernel checkout fdf4bb7fcc984811cef48ce817d89aac064b984a
+   ```
+3. CK headers are generated at CK configure time (`ck/config.h`,
+   `ck/version.h`). The llama.cpp build includes the CK source tree, so run
+   CK's configure once and copy the headers into `include/ck/`:
+   ```
+   cmake -S composable_kernel -B build-ck -DAMDGPU_TARGETS=gfx1151 -DCK_ENABLE_WMMA=ON
+   cp build-ck/include/ck/config.h build-ck/include/ck/version.h composable_kernel/include/ck/
+   ```
+4. Configure the llama.cpp build against the venv ROCm 7.15 SDK:
+   ```
+   export ROCM_PATH=$SDK/_rocm_sdk_devel
+   cmake -S ROCmFPX -B build-promptforge -G Ninja \
+     -DCMAKE_BUILD_TYPE=Release \
+     -DCMAKE_C_COMPILER=$ROCM_PATH/llvm/bin/clang \
+     -DCMAKE_CXX_COMPILER=$ROCM_PATH/llvm/bin/clang++ \
+     -DCMAKE_HIP_COMPILER=$ROCM_PATH/llvm/bin/clang++ \
+     -DCMAKE_PREFIX_PATH=$ROCM_PATH \
+     -DGGML_HIP=ON -DGGML_CUDA=OFF -DGGML_VULKAN=OFF \
+     -DGGML_HIP_FORCE_MMQ=ON -DGGML_HIP_GRAPHS=ON \
+     -DGGML_HIP_MMQ_MFMA=ON -DGGML_HIP_NO_VMM=ON \
+     -DGGML_HIP_ROCWMMA_FATTN=OFF -DGGML_NATIVE=ON \
+     -DAMDGPU_TARGETS=gfx1151 -DGPU_BUILD_TARGETS=gfx1151 \
+     -DPROMPTFORGE_CK_ROOT=$CK_ROOT \
+     -DLLAMA_BUILD_SERVER=ON \
+     -DLLAMA_BUILD_WEBUI=OFF
+   cmake --build build-promptforge --target llama-server -j$(nproc)
+   ```
+   `LLAMA_BUILD_WEBUI=OFF` is required. The embedded browser UI downloads
+   assets from an HF bucket (`llama-ui`) that returns 404; the service is
+   consumed through the OpenAI-compatible API, so the UI is not needed.
+
+Build output: `build-promptforge/bin/llama-server`.
+
+```mermaid
+graph TD
+    A[Clone ROCmFPX v2.3 + CK pinned] --> B[Configure CK, copy generated headers]
+    B --> C[Configure llama-server vs ROCm 7.15 SDK]
+    C --> D[Build llama-server]
+    D --> E[Verify --list-devices shows gfx1151]
+    E --> F[Download v2 GGUF + FFN + GDN]
+    F --> G[Deploy frank-default-small]
+    G --> H[Health check + chat completion]
+```
+
+## Model Download
+
+Downloaded with the HF token to
+`/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX/`.
+
+## Service
+
+- Unit: `frank-default-small.service`, port 52625
+- Script: `scripts/serve-frank-small.sh`, env `scripts/frank-default-small.env`
+- The script exports the ROCm 7.15 SDK libraries in `LD_LIBRARY_PATH` before
+  the system ROCm. `ROCM_PATH` points at the devel SDK.
+
+### Launch profile (from the model card)
+
+- 64K context, one slot, f16 KV, context checkpoints (32), prompt cache
+- Native MTP draft, depth 4, device ROCm0
+- `--temp 0.7 --top-p 0.95`, reasoning off
+- PromptForge env: `PROMPTFORGE_MODE=m2048_fused_tail1476`,
+  `GGML_CUDA_GRAPH_OPT=0`, `LLAMA_MTP_CPU_ARGMAX_FASTPATH=1`
+
+### Required environment
+
+```
+export PROMPTFORGE_SIDECAR=...FFN.pfs
+export PROMPTFORGE_GDN_SIDECAR=...GDN.pfs
+export PROMPTFORGE_MODE=m2048_fused_tail1476
+export GGML_CUDA_GRAPH_OPT=0
+export LLAMA_MTP_CPU_ARGMAX_FASTPATH=1
+```
+
+The `p_min=0` MTP profile uses the CPU argmax fast path. This avoids ROCm's
+unsupported TOP_K operator.
+
+## Operational Notes
+
+- Health: `curl http://127.0.0.1:52625/health`. Returns 503 while loading.
+- Model load is slow (34.9 GiB payload); do not kill the process during
+  warmup.
+- `deploy.sh dankclaw` waits 20 minutes for health.
+- The systemd unit must not set `HOST` or `TEMP` env vars (HIP collision
+  rule). `--host`/`--port`/`--temp` are CLI flags only.
+
+## See Also
+
+- [[frank-redo-2026-08-17]] — the fleet topologies this slot has served.
+- [[Shared Memory Architecture Dankclaw]] — unified memory sizing.
+- [[qwen3.8-strix-optimizations]] — llama.cpp ROCm0 baselines on this iGPU.
+- [[VLLM Overview]] — vLLM reference (retired for this slot).

docs/index.md

@@ -1,8 +1,8 @@
 # vLLM Scripts — Knowledge Base
 
 Documentation for the LLM inference fleet on two local machines
-(2026-08-17 redo): llama.cpp FastMTP on Dankserver and vLLM ROCm on
+(2026-08-18): llama.cpp FastMTP on Dankserver and CIRU ROCmFPX llama.cpp
-Dankclaw.
+on Dankclaw.
 
 ## Deployment Topology
 
@@ -19,10 +19,10 @@ FastMTP 32K draft + ngram-mod, 128K ctx]
 
     subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
         D[gfx1151 iGPU unified memory]
-        E[vLLM ROCm
+        E[CIRU ROCmFPX llama.cpp
 frank-default-small, port 52625]
-        F[nameistoken Qwen3.6-35B-A3B Quark W8A8 INT8
+        F[jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge
-full 262144 ctx, ngram spec + tools, 4 seqs]
+64K ctx, native MTP, PromptForge DualView]
         D --> E --> F
     end
 
@@ -40,17 +40,23 @@ full 262144 ctx, ngram spec + tools, 4 seqs]
 ## Active Deployment
 
 - [[deployment]] — deploy and update process for both servers
-- [[frank-redo-2026-08-17]] — the full redo: topology change, model
+- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
-  choices, context math, and the two install runbooks
+- [[ciru-activefpx-promptforge]] — the CIRU runtime on Dankclaw
+  (frank-default-small)
 - [[Dankserver]] — the llama.cpp FastMTP box (frank-default-large)
-- [[Dankclaw]] — the vLLM ROCm box (frank-default-small)
+- [[Dankclaw]] — the CIRU ROCmFPX box (frank-default-small)
 
-## vLLM Reference (Dankclaw)
+## CIRU Reference (Dankclaw)
+
+- [[ciru-activefpx-promptforge]] — model, ROCm 7.15 toolchain, build,
+  and the frank-default-small service
+
+## vLLM Reference (historical)
 
 - [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism
 - [[VLLM Index]] — quick reference with commands, endpoints, and parameter table
 - [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware
-  notes and sizing (the model family served by frank-default-small)
+  notes and sizing (the model family formerly served by frank-default-small)
 
 ## Memory Management
 

scripts/deploy.sh

@@ -16,9 +16,9 @@
 #   ./deploy.sh <target> [--dry-run]
 #   FRANK_SUDO_PW=... ./deploy.sh <target>
 #
-# Targets (2026-08-17 topology):
+# Targets (2026-08-18 topology):
 #   dankserver - llama.cpp FastMTP (frank-default-large.service, port 7770)
-#   dankclaw   - vLLM ROCm (frank-default-small.service, port 52625)
+#   dankclaw   - CIRU ROCmFPX llama.cpp (frank-default-small.service, port 52625)
 set -euo pipefail
 
 # ---------------------------------------------------------------------------
@@ -41,7 +41,8 @@ REMOTE_DIR[dankclaw]="/home/dank/vllmScripts"
 SERVICE[dankclaw]="frank-default-small"
 SERVICE_HEALTH[dankclaw]="http://127.0.0.1:52625/health"
 DROP_CACHES[dankclaw]=0
-# vLLM loads a 38 GB checkpoint; allow 20 minutes.
+# CIRU llama.cpp (ROCm 7.15) loads the 34.9 GiB ActiveFPX payload + MTP
+# draft and initializes PromptForge views; allow 20 minutes.
 HEALTH_WAIT[dankclaw]=600
 
 TARGET="${1:-}"

scripts/frank-default-small.env

@@ -1,72 +1,65 @@
-# Tunables for frank-default-small (vLLM, dankclaw, Strix Halo gfx1151).
+# Tunables for frank-default-small (CIRU llama.cpp, dankclaw, Strix Halo
+# gfx1151).
 #
-# Runtime: vLLM (ROCm) in /home/dank/vllm-venv serving
+# Runtime: the PromptForge llama-server built from ciru-ai/ROCmFPX tag
-# Avesed/Qwen3.6-35B-A3B-INT8-W8A8 (compressed-tensors INT8 W8A8, MoE 35B
+# qwen3.8-activefpx-promptforge-v2.3 serving
-# total / 3B active, Qwen3.5-MoE hybrid: 30 of 40 layers are GatedDeltaNet
+# jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge (v2). ActiveFPX compact
-# linear attention). Alias frank-default-small, port 52625.
+# GGUF with PromptForge companions for CIRU DualView prompt acceleration and
+# native depth-4 MTP. Alias frank-default-small, port 52625.
 #
-# The ROCm vLLM install was created with:
+# This runtime replaced the retired vLLM service that previously occupied
-#   uv venv --python /usr/bin/python3.12 /home/dank/vllm-venv
+# this slot. See docs/ciru-activefpx-promptforge.md.
-#   uv pip install --python /home/dank/vllm-venv vllm --torch-backend=auto
+#
-# The --torch-backend=auto resolver pulls the AMD ROCm wheels (torch +
+# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The ROCm
-# amd-torch-device-gfx1151 + vllm rocm variant) from the AMD index.
+# HIP runtime reads those bare env vars (see the 2026-08-13 TEMP incident).
+# llama.cpp reads --host/--port/--temp as CLI flags, never as env vars.
+
+# CIRU llama-server binary (ROCmFPX v2.3 build, see docs/ciru runtime page).
+# Built against the private TheRock ROCm 7.15 SDK kept in the retired
+# ling3-ciru venv. ROCm 7.15 is required for the advertised Strix Halo
+# performance; the system ROCm 7.2.4 selects slower/fallback kernels.
+LLAMA_BIN=/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server
+
+# Model files. The three-file payload (v2 GGUF + FFN + GDN) enables CIRU
+# DualView prompt acceleration; omit the two .pfs files to run the GGUF alone
+# and save ~19.7 GiB of memory at the cost of prompt throughput.
+MODEL_DIR=/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX
+MODEL_FILE=Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-v2.gguf
+FFN_PFS=Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-FFN.pfs
+GDN_PFS=Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-GDN.pfs
+
+# ROCm 7.15 SDK paths (inside the retired ling3-ciru venv). The serve script
+# exports ROCM_PATH and LD_LIBRARY_PATH from these so the 7.15 libraries
+# precede any system ROCm 7.2.4.
+ROCM_DEVEL=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_devel
+ROCM_LIB=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_libraries
+ROCM_CORE=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_core
 
-VLLM_BIN=/home/dank/vllm-venv/bin/vllm
-# 2026-08-17 (evening): swapped from Avesed/Qwen3.6-35B-A3B-INT8-W8A8 to
-# nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8 (AMD Quark W8A8, same
-# architecture, vLLM-native expert layout). The Avesed quant's tool-call
-# XML fidelity was poor (malformed <function> tags); the Quark model
-# follows the Qwen3 XML tool format cleanly.
-MODEL=nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8
 ALIAS=frank-default-small
 BIND_ADDR=0.0.0.0
 LISTEN_PORT=52625
 
-# Full native context. The KV cache stays unquantized (auto = model dtype,
+# Serving defaults from the model card: 64K context, single slot, f16 KV,
-# bf16). Worst-case KV at 262144 x 4 sequences is ~10.7 GiB; weights are
+# context checkpoints, prompt cache. Native MTP depth-4.
-# ~35 GiB; total fits the Strix Halo unified pool with room to spare.
+CTX_SIZE=65536
-MAX_MODEL_LEN=262144
+PARALLEL=1
-KV_CACHE_DTYPE=auto
+BATCH_SIZE=2048
+UBATCH_SIZE=2048
+CACHE_TYPE_K=f16
+CACHE_TYPE_V=f16
+CTX_CHECKPOINTS=32
+CACHE_RAM=8192
 
-# Concurrency and batching (user request). max-num-batched-tokens lowered
+# Native MTP draft (depth-4, greedy, ROCm-safe CPU argmax fast path).
-# to 8192 on 2026-08-17 (evening). vLLM has no separate ubatch knob.
+SPEC_DRAFT_N_MAX=4
-MAX_NUM_SEQS=4
+SPEC_DRAFT_N_MIN=0
-MAX_NUM_BATCHED_TOKENS=8192
+SPEC_DRAFT_P_MIN=0.0
+SPEC_DRAFT_P_SPLIT=0.10
 
-# vLLM has no separate ubatch knob. The scheduler issues one batch per step
+# Sampling (model card defaults).
-# with up to max-num-batched-tokens tokens; the KV cache is preallocated at
+TEMPERATURE=0.7
-# max-model-len. Set gpu-memory-utilization below 1.0 so the unified pool
+TOP_P=0.95
-# keeps headroom for the host.
-GPU_MEMORY_UTILIZATION=0.92
 
-# Prefix caching. This model is hybrid (GatedDeltaNet + full attention); vLLM
+REASONING=off
-# keeps prefix caching OPT-IN by default for hybrid models ("while the feature
-# matures"), so it is off unless we ask for it. Turning it on reuses the KV of
-# the shared prompt prefix (pi re-sends the same system prompt + tool schema
-# every turn), so follow-up requests skip re-prefilling thousands of tokens.
-# Big TTFT win for the agent loop. Set 0 to disable.
-ENABLE_PREFIX_CACHING=1
 
-# ngram speculative decoding (vLLM ngram proposer).
+VERBOSITY=4
-# prompt_lookup_max = ngram window; num_speculative_tokens = draft depth.
-NGRAM_LOOKUP_MAX=5
-NGRAM_LOOKUP_MIN=1
-NUM_SPEC_TOKENS=5
-
-# Tool calling. Clients (pi, OpenWebUI) send tool_choice=auto; without
-# --enable-auto-tool-choice vLLM rejects it with HTTP 400. The Avesed
-# chat template renders tools in the Qwen3 XML format, so the qwen3_xml
-# parser is the matching one. CHAT_TEMPLATE must point at the repo's
-# chat_template.jinja (the tokenizer_config.json ships none; vLLM's
-# built-in template does not render the XML tool instructions).
-ENABLE_AUTO_TOOL_CHOICE=1
-TOOL_CALL_PARSER=qwen3_xml
-CHAT_TEMPLATE=/home/dank/models/qwen36-chat_template-tools.jinja
-
-# RDNA flash-attention for the vision encoder (and general FA paths).
-# Without this, the mm encoder falls back to torch SDPA and the warmup
-# OOMs (256 GiB score matrix at max_model_len; measured 2026-08-17).
-# Requires the flash_attn package with flash_attn_triton_amd findable
-# (the pure-Python copy from the CIRU package plus a flash_attn_2_cuda
-# stub; see docs/frank-redo-2026-08-17.md).
-FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE

scripts/frank-default-small.service

@@ -1,5 +1,5 @@
 [Unit]
-Description=vLLM Qwen3.6-35B-A3B INT8 W8A8 ngram spec (dankclaw, frank-default-small)
+Description=CIRU llama.cpp Qwen3.8-27B ActiveFPX PromptForge ngram MTP (dankclaw, frank-default-small)
 After=network-online.target
 Wants=network-online.target
 
@@ -8,7 +8,6 @@ Type=simple
 User=dank
 Group=dank
 Environment="FRANK_SMALL_ENV_FILE=/home/dank/vllmScripts/scripts/frank-default-small.env"
-Environment="HF_HOME=/home/dank/.cache/huggingface"
 ExecStart=/home/dank/vllmScripts/scripts/serve-frank-small.sh
 Restart=on-failure
 RestartSec=10

scripts/serve-frank-small.sh

@@ -1,10 +1,16 @@
 #!/usr/bin/env bash
-# Serve Avesed/Qwen3.6-35B-A3B-INT8-W8A8 via vLLM on Dankclaw (Strix Halo).
+# Serve jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge via the CIRU ROCmFPX
+# llama.cpp runtime on Dankclaw (Strix Halo gfx1151).
 #
-# Runs the ROCm vLLM build in /home/dank/vllm-venv. The model is a
+# Runtime: the PromptForge llama-server built from ciru-ai/ROCmFPX tag
-# compressed-tensors INT8 W8A8 quant (MoE, 35B total / 3B active). Full
+# qwen3.8-activefpx-promptforge-v2.3 (commit 103af8b41) against the private
-# 262144 context, unquantized KV cache, 4 concurrent sequences, ngram
+# TheRock ROCm 7.15 SDK (kept in the retired ling3-ciru venv). The model is
-# speculative decoding.
+# a GGUF with optional PromptForge companion views (FFN/GDN .pfs) that enable
+# CIRU DualView prompt acceleration. Native depth-4 MTP speculative decoding.
+#
+# The model card requires ROCm 7.15 (not the system ROCm 7.2.4) for the
+# advertised Strix Halo performance. LD_LIBRARY_PATH therefore points at the
+# 7.15 SDK libraries first. An older system ROCm must not precede them.
 #
 # Tunables come from frank-default-small.env in this directory unless
 # FRANK_SMALL_ENV_FILE overrides that path.
@@ -20,55 +26,93 @@ if [[ -f "$ENV_FILE" ]]; then
 fi
 
 # --- defaults (used when the env file does not set them) -------------------
-: "${VLLM_BIN:=/home/dank/vllm-venv/bin/vllm}"
+: "${LLAMA_BIN:=/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server}"
-: "${MODEL:=nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8}"
+: "${MODEL_DIR:=/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX}"
+: "${MODEL_FILE:=Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-v2.gguf}"
+: "${FFN_PFS:=Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-FFN.pfs}"
+: "${GDN_PFS:=Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-GDN.pfs}"
+: "${ROCM_DEVEL:=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_devel}"
+: "${ROCM_LIB:=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_libraries}"
+: "${ROCM_CORE:=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_core}"
 : "${ALIAS:=frank-default-small}"
 : "${BIND_ADDR:=0.0.0.0}"
 : "${LISTEN_PORT:=52625}"
-: "${MAX_MODEL_LEN:=262144}"
+: "${CTX_SIZE:=65536}"
-: "${KV_CACHE_DTYPE:=auto}"
+: "${PARALLEL:=1}"
-: "${MAX_NUM_SEQS:=4}"
+: "${BATCH_SIZE:=2048}"
-: "${MAX_NUM_BATCHED_TOKENS:=8192}"
+: "${UBATCH_SIZE:=2048}"
-: "${GPU_MEMORY_UTILIZATION:=0.92}"
+: "${CACHE_TYPE_K:=f16}"
-: "${NGRAM_LOOKUP_MAX:=5}"
+: "${CACHE_TYPE_V:=f16}"
-: "${NGRAM_LOOKUP_MIN:=1}"
+: "${CTX_CHECKPOINTS:=32}"
-: "${NUM_SPEC_TOKENS:=5}"
+: "${CACHE_RAM:=8192}"
-: "${ENABLE_AUTO_TOOL_CHOICE:=1}"
+: "${SPEC_DRAFT_N_MAX:=4}"
-: "${TOOL_CALL_PARSER:=qwen3_xml}"
+: "${SPEC_DRAFT_N_MIN:=0}"
-: "${ENABLE_PREFIX_CACHING:=1}"
+: "${SPEC_DRAFT_P_MIN:=0.0}"
-# The Avesed tokenizer_config.json has no chat_template; vLLM would use its
+: "${SPEC_DRAFT_P_SPLIT:=0.10}"
-# built-in template which does not render the model's XML tool format. Use
+: "${TEMPERATURE:=0.7}"
-# the repo's chat_template.jinja (ships the <tool_call> instructions).
+: "${TOP_P:=0.95}"
-: "${CHAT_TEMPLATE:=/home/dank/models/qwen36-chat_template-tools.jinja}"
+: "${REASONING:=off}"
+: "${VERBOSITY:=4}"
 
-if [[ ! -x "$VLLM_BIN" ]]; then
+MODEL="$MODEL_DIR/$MODEL_FILE"
-    echo "missing vllm binary: $VLLM_BIN" >&2
+export PROMPTFORGE_SIDECAR="$MODEL_DIR/$FFN_PFS"
+export PROMPTFORGE_GDN_SIDECAR="$MODEL_DIR/$GDN_PFS"
+export PROMPTFORGE_MODE="m2048_fused_tail1476"
+export GGML_CUDA_GRAPH_OPT=0
+export LLAMA_MTP_CPU_ARGMAX_FASTPATH=1
+
+if [[ ! -f "$MODEL" ]]; then
+    echo "missing required file: $MODEL" >&2
+    exit 1
+fi
+if [[ ! -f "$PROMPTFORGE_SIDECAR" ]]; then
+    echo "missing PromptForge FFN companion: $PROMPTFORGE_SIDECAR" >&2
+    exit 1
+fi
+if [[ ! -f "$PROMPTFORGE_GDN_SIDECAR" ]]; then
+    echo "missing PromptForge GDN companion: $PROMPTFORGE_GDN_SIDECAR" >&2
+    exit 1
+fi
+if [[ ! -x "$LLAMA_BIN" ]]; then
+    echo "missing llama-server binary: $LLAMA_BIN" >&2
     exit 1
 fi
 
-TOOL_ARGS=()
+# ROCm 7.15 SDK libraries must precede any system ROCm in LD_LIBRARY_PATH.
-if [[ "${ENABLE_AUTO_TOOL_CHOICE}" == "1" ]]; then
+export ROCM_PATH="$ROCM_DEVEL"
-    TOOL_ARGS+=(--enable-auto-tool-choice)
+export LD_LIBRARY_PATH="${ROCM_LIB}/lib:${ROCM_CORE}/lib:${ROCM_DEVEL}/lib${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}"
-    TOOL_ARGS+=(--tool-call-parser "${TOOL_CALL_PARSER}")
-fi
 
-PC_ARGS=()
+exec "$LLAMA_BIN" \
-if [[ "${ENABLE_PREFIX_CACHING}" == "1" ]]; then
+    --model "$MODEL" \
-    PC_ARGS+=(--enable-prefix-caching)
+    --alias "$ALIAS" \
-fi
-
-exec "$VLLM_BIN" serve "$MODEL" \
     --host "$BIND_ADDR" \
     --port "$LISTEN_PORT" \
-    --served-model-name "$ALIAS" \
+    -dev ROCm0 \
-    --max-model-len "$MAX_MODEL_LEN" \
+    -ngl 999 \
-    --kv-cache-dtype "$KV_CACHE_DTYPE" \
+    -c "$CTX_SIZE" \
-    --max-num-seqs "$MAX_NUM_SEQS" \
+    -b "$BATCH_SIZE" \
-    --max-num-batched-tokens "$MAX_NUM_BATCHED_TOKENS" \
+    -ub "$UBATCH_SIZE" \
-    --gpu-memory-utilization "$GPU_MEMORY_UTILIZATION" \
+    -fa on \
-    --speculative-config "{\"method\":\"ngram\",\"prompt_lookup_max\":$NGRAM_LOOKUP_MAX,\"prompt_lookup_min\":$NGRAM_LOOKUP_MIN,\"num_speculative_tokens\":$NUM_SPEC_TOKENS}" \
+    -ctk "$CACHE_TYPE_K" \
-    --reasoning-parser qwen3 \
+    -ctv "$CACHE_TYPE_V" \
-    --trust-remote-code \
+    -np "$PARALLEL" \
-    --chat-template "$CHAT_TEMPLATE" \
+    -ctxcp "$CTX_CHECKPOINTS" \
-    "${PC_ARGS[@]}" \
+    --cache-ram "$CACHE_RAM" \
-    "${TOOL_ARGS[@]}"
+    --cache-prompt \
+    --metrics \
+    --spec-type draft-mtp \
+    --spec-draft-device ROCm0 \
+    --spec-draft-ngl 999 \
+    --spec-draft-type-k f16 \
+    --spec-draft-type-v f16 \
+    --spec-draft-n-max "$SPEC_DRAFT_N_MAX" \
+    --spec-draft-n-min "$SPEC_DRAFT_N_MIN" \
+    --spec-draft-p-min "$SPEC_DRAFT_P_MIN" \
+    --spec-draft-p-split "$SPEC_DRAFT_P_SPLIT" \
+    --spec-draft-backend-sampling \
+    --temp "$TEMPERATURE" \
+    --top-p "$TOP_P" \
+    --reasoning "$REASONING" \
+    --reasoning-format none \
+    --reasoning-budget -1 \
+    -lv "$VERBOSITY"