dankclaw: replace vLLM with CIRU ROCmFPX llama.cpp runtime

Retire the vLLM service (frank-default-small, port 52625) on Dankclaw and serve jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge through the CIRU ROCmFPX llama.cpp runtime instead. - Build llama-server from ciru-ai/ROCmFPX tag qwen3.8-activefpx-promptforge-v2.3 (commit 103af8b41) against the private TheRock ROCm 7.15 SDK in the retired ling3-ciru venv. The model card requires ROCm 7.15 for the advertised Strix Halo performance; the system ROCm 7.2.4 selects slower kernels. CK pinned at fdf4bb7fcc984811cef48ce817d89aac064b984a; generated ck/config.h + ck/version.h copied into the CK source tree so the ggml-hip build can include them. LLAMA_BUILD_WEBUI=OFF (HF bucket llama-ui returns 404; server is consumed via the OpenAI API). - New scripts/serve-frank-small.sh + frank-default-small.env: 64K ctx, one slot, f16 KV, context checkpoints, prompt cache, native depth-4 MTP on ROCm0, PromptForge DualView sidecars (FFN + GDN), temp 0.7 top-p 0.95, reasoning off. LD_LIBRARY_PATH puts the 7.15 SDK libs before any system ROCm. - Update the systemd unit description (same service name and port). - deploy.sh: update the dankclaw target comment. - Docs: new docs/ciru-activefpx-promptforge.md with the build runbook; update docs/index.md and AGENTS.md topology. - The vLLM venvs (vllm-venv, vllm-vk-venv) were moved to /home/dank/.AGENTTRASH/ on the target; the vllm-vulkan plugin and docs were removed from the repo in an earlier commit.
2026-08-17 21:12:35 -07:00
1 parent 4941d911f2
commit 3b51b5a94a
@@ -1,72 +1,65 @@
-# Tunables for frank-default-small (vLLM, dankclaw, Strix Halo gfx1151).
+# Tunables for frank-default-small (CIRU llama.cpp, dankclaw, Strix Halo
+# gfx1151).
 #
-# Runtime: vLLM (ROCm) in /home/dank/vllm-venv serving
-# Avesed/Qwen3.6-35B-A3B-INT8-W8A8 (compressed-tensors INT8 W8A8, MoE 35B
-# total / 3B active, Qwen3.5-MoE hybrid: 30 of 40 layers are GatedDeltaNet
-# linear attention). Alias frank-default-small, port 52625.
+# Runtime: the PromptForge llama-server built from ciru-ai/ROCmFPX tag
+# qwen3.8-activefpx-promptforge-v2.3 serving
+# jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge (v2). ActiveFPX compact
+# GGUF with PromptForge companions for CIRU DualView prompt acceleration and
+# native depth-4 MTP. Alias frank-default-small, port 52625.
 #
-# The ROCm vLLM install was created with:
-#   uv venv --python /usr/bin/python3.12 /home/dank/vllm-venv
-#   uv pip install --python /home/dank/vllm-venv vllm --torch-backend=auto
-# The --torch-backend=auto resolver pulls the AMD ROCm wheels (torch +
-# amd-torch-device-gfx1151 + vllm rocm variant) from the AMD index.
+# This runtime replaced the retired vLLM service that previously occupied
+# this slot. See docs/ciru-activefpx-promptforge.md.
+#
+# WARNING: do not name any variable TEMP, HOST, or GPU_MAX_HEAP_SIZE. The ROCm
+# HIP runtime reads those bare env vars (see the 2026-08-13 TEMP incident).
+# llama.cpp reads --host/--port/--temp as CLI flags, never as env vars.
+
+# CIRU llama-server binary (ROCmFPX v2.3 build, see docs/ciru runtime page).
+# Built against the private TheRock ROCm 7.15 SDK kept in the retired
+# ling3-ciru venv. ROCm 7.15 is required for the advertised Strix Halo
+# performance; the system ROCm 7.2.4 selects slower/fallback kernels.
+LLAMA_BIN=/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server
+
+# Model files. The three-file payload (v2 GGUF + FFN + GDN) enables CIRU
+# DualView prompt acceleration; omit the two .pfs files to run the GGUF alone
+# and save ~19.7 GiB of memory at the cost of prompt throughput.
+MODEL_DIR=/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX
+MODEL_FILE=Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-v2.gguf
+FFN_PFS=Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-FFN.pfs
+GDN_PFS=Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-GDN.pfs
+
+# ROCm 7.15 SDK paths (inside the retired ling3-ciru venv). The serve script
+# exports ROCM_PATH and LD_LIBRARY_PATH from these so the 7.15 libraries
+# precede any system ROCm 7.2.4.
+ROCM_DEVEL=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_devel
+ROCM_LIB=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_libraries
+ROCM_CORE=/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/_rocm_sdk_core

-VLLM_BIN=/home/dank/vllm-venv/bin/vllm
-# 2026-08-17 (evening): swapped from Avesed/Qwen3.6-35B-A3B-INT8-W8A8 to
-# nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8 (AMD Quark W8A8, same
-# architecture, vLLM-native expert layout). The Avesed quant's tool-call
-# XML fidelity was poor (malformed <function> tags); the Quark model
-# follows the Qwen3 XML tool format cleanly.
-MODEL=nameistoken/Qwen3.6-35B-A3B-Quark-W8A8-INT8
 ALIAS=frank-default-small
 BIND_ADDR=0.0.0.0
 LISTEN_PORT=52625

-# Full native context. The KV cache stays unquantized (auto = model dtype,
-# bf16). Worst-case KV at 262144 x 4 sequences is ~10.7 GiB; weights are
-# ~35 GiB; total fits the Strix Halo unified pool with room to spare.
-MAX_MODEL_LEN=262144
-KV_CACHE_DTYPE=auto
+# Serving defaults from the model card: 64K context, single slot, f16 KV,
+# context checkpoints, prompt cache. Native MTP depth-4.
+CTX_SIZE=65536
+PARALLEL=1
+BATCH_SIZE=2048
+UBATCH_SIZE=2048
+CACHE_TYPE_K=f16
+CACHE_TYPE_V=f16
+CTX_CHECKPOINTS=32
+CACHE_RAM=8192

-# Concurrency and batching (user request). max-num-batched-tokens lowered
-# to 8192 on 2026-08-17 (evening). vLLM has no separate ubatch knob.
-MAX_NUM_SEQS=4
-MAX_NUM_BATCHED_TOKENS=8192
+# Native MTP draft (depth-4, greedy, ROCm-safe CPU argmax fast path).
+SPEC_DRAFT_N_MAX=4
+SPEC_DRAFT_N_MIN=0
+SPEC_DRAFT_P_MIN=0.0
+SPEC_DRAFT_P_SPLIT=0.10

-# vLLM has no separate ubatch knob. The scheduler issues one batch per step
-# with up to max-num-batched-tokens tokens; the KV cache is preallocated at
-# max-model-len. Set gpu-memory-utilization below 1.0 so the unified pool
-# keeps headroom for the host.
-GPU_MEMORY_UTILIZATION=0.92
+# Sampling (model card defaults).
+TEMPERATURE=0.7
+TOP_P=0.95

-# Prefix caching. This model is hybrid (GatedDeltaNet + full attention); vLLM
-# keeps prefix caching OPT-IN by default for hybrid models ("while the feature
-# matures"), so it is off unless we ask for it. Turning it on reuses the KV of
-# the shared prompt prefix (pi re-sends the same system prompt + tool schema
-# every turn), so follow-up requests skip re-prefilling thousands of tokens.
-# Big TTFT win for the agent loop. Set 0 to disable.
-ENABLE_PREFIX_CACHING=1
+REASONING=off

-# ngram speculative decoding (vLLM ngram proposer).
-# prompt_lookup_max = ngram window; num_speculative_tokens = draft depth.
-NGRAM_LOOKUP_MAX=5
-NGRAM_LOOKUP_MIN=1
-NUM_SPEC_TOKENS=5
-
-# Tool calling. Clients (pi, OpenWebUI) send tool_choice=auto; without
-# --enable-auto-tool-choice vLLM rejects it with HTTP 400. The Avesed
-# chat template renders tools in the Qwen3 XML format, so the qwen3_xml
-# parser is the matching one. CHAT_TEMPLATE must point at the repo's
-# chat_template.jinja (the tokenizer_config.json ships none; vLLM's
-# built-in template does not render the XML tool instructions).
-ENABLE_AUTO_TOOL_CHOICE=1
-TOOL_CALL_PARSER=qwen3_xml
-CHAT_TEMPLATE=/home/dank/models/qwen36-chat_template-tools.jinja
-
-# RDNA flash-attention for the vision encoder (and general FA paths).
-# Without this, the mm encoder falls back to torch SDPA and the warmup
-# OOMs (256 GiB score matrix at max_model_len; measured 2026-08-17).
-# Requires the flash_attn package with flash_attn_triton_amd findable
-# (the pure-Python copy from the CIRU package plus a flash_attn_2_cuda
-# stub; see docs/frank-redo-2026-08-17.md).
-FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE
+VERBOSITY=4