dank/serverAppConfigs

Fork 0

Files

T

Frank c02d67513d Update benchmark documentation after harness replacement

2026-08-25 19:08:42 -07:00

12 KiB

Raw Permalink Blame History

type, title, description, tags, timestamp

type

title

description

CIRU ActiveFPX PromptForge on Dankclaw

Active 2026-08-22. frank-default-large serves this v3 model again after a same-day Kairic Edge IU4 experiment was reverted (operator decision). Commit c60156f restored the v3 script pair. See kairic-edge for the retired experiment reference.

frank-default-large (port 7770, moved from 52625 on 2026-08-22) serves jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v3 through the CIRU ROCmFPX llama.cpp runtime. This replaced the retired vLLM service that previously occupied the slot. The v2 deployment moved to v3 on 2026-08-21.

What This Is

The model is a GGUF, not a vLLM checkpoint. v3 is the consolidated quality-and-speed release of Qwen3.8-27B for AMD ROCm: an FFN-equalized compact body, 50 retained late-writer Q6 tensors, full-Q8 output projection, full-W8 PromptForge view (FFN + Gated DeltaNet), the small-row W8 route, a draft-only Output-K8 shortlist proxy with Q8 reranking, native depth-4 MTP (no ngram), and the BF16 Qwen3.8 vision tower + 5120-wide merger as a separate projector.

Artifact	Size
`Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-v3-Q8-FFNEQ-LateQ6.gguf`	15.44 GiB
`Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-FFN.pfs`	15.95 GiB
`Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-GDN.pfs`	3.75 GiB
`Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-Output-K8.pfs`	0.67 GiB
`Qwen3.8-27B-mmproj-BF16.mmproj`	0.87 GiB
Full v3 payload (GGUF + 3 .pfs + mmproj)	~36.7 GiB

The .pfs companions and the Output-K8 proxy are optional. Omit them to run the GGUF alone and save ~20.6 GiB of memory. The GGUF remains independently usable.

Requirements

ROCm 7.15 is required for the advertised Strix Halo performance. The system ROCm (/opt/rocm-7.2.4) selects slower or fallback kernels.
The pinned runtime is ciru-ai/ROCmFPX tag qwen3.8-activefpx-promptforge-v2.3 (commit 103af8b41) plus the v3 runtime patch runtime/qwen38-v3-output-k8-runtime.patch. On Dankclaw the patch is commit af0eac6 on top of the v2.3 checkout.
Composable Kernel at fdf4bb7fcc984811cef48ce817d89aac064b984a.
Validated GPU: AMD Radeon 8060S / gfx1151.

ROCm 7.15 Toolchain

The retired ling3-ciru package left a complete TheRock ROCm 7.15 SDK (7.15.0a20260728) inside its venv:

/home/dank/ling3-ciru/Ling-3.0-Flash-CIRU-int4-Strix-native/.venv/lib/python3.12/site-packages/
  _rocm_sdk_devel/      # hipcc, amdclang 23, cmake, headers
  _rocm_sdk_libraries/  # hipblas, hipblaslt, rocroller
  _rocm_sdk_core/       # libamdhip64, libhsa-runtime64
  rocm-sdk-device-gfx1151

The compiler is amdclang 23. rocm-sdk version reports 7.15.0a20260728. The binary is built against and run with these 7.15 libraries.

Build Runbook (2026-08-18, v2.3; +v3 patch on 2026-08-21)

Step 1-4 below built v2.3 on 2026-08-18. On 2026-08-21 the v3 Output-K8 runtime patch was applied on top and the server rebuilt:

Apply the v3 runtime patch to the v2.3 checkout and rebuild:

% verify the patch hash: 8ba7d721ea380b060aeb0c7595b8b05f1582c0f7bf7a4b89e0910bc8e72465d5
git am runtime/qwen38-v3-output-k8-runtime.patch   # -> commit af0eac6
cmake --build build-promptforge --target llama-server -j$(nproc)

The pre-built binary is kept as build-promptforge/bin/llama-server.v2.3-bak for rollback.

Install host build tools (ninja-build was missing): sudo apt install ninja-build.

Clone the runtime and CK at their pinned revisions:

git clone --depth 1 --branch qwen3.8-activefpx-promptforge-v2.3 \
  https://github.com/ciru-ai/ROCmFPX.git
git clone --depth 1 https://github.com/ROCm/composable_kernel.git
git -C composable_kernel checkout fdf4bb7fcc984811cef48ce817d89aac064b984a

CK headers are generated at CK configure time (ck/config.h, ck/version.h). The llama.cpp build includes the CK source tree, so run CK's configure once and copy the headers into include/ck/:
```
cmake -S composable_kernel -B build-ck -DAMDGPU_TARGETS=gfx1151 -DCK_ENABLE_WMMA=ON
cp build-ck/include/ck/config.h build-ck/include/ck/version.h composable_kernel/include/ck/
```

Configure the llama.cpp build against the venv ROCm 7.15 SDK:

export ROCM_PATH=$SDK/_rocm_sdk_devel
cmake -S ROCmFPX -B build-promptforge -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_C_COMPILER=$ROCM_PATH/llvm/bin/clang \
  -DCMAKE_CXX_COMPILER=$ROCM_PATH/llvm/bin/clang++ \
  -DCMAKE_HIP_COMPILER=$ROCM_PATH/llvm/bin/clang++ \
  -DCMAKE_PREFIX_PATH=$ROCM_PATH \
  -DGGML_HIP=ON -DGGML_CUDA=OFF -DGGML_VULKAN=OFF \
  -DGGML_HIP_FORCE_MMQ=ON -DGGML_HIP_GRAPHS=ON \
  -DGGML_HIP_MMQ_MFMA=ON -DGGML_HIP_NO_VMM=ON \
  -DGGML_HIP_ROCWMMA_FATTN=OFF -DGGML_NATIVE=ON \
  -DAMDGPU_TARGETS=gfx1151 -DGPU_BUILD_TARGETS=gfx1151 \
  -DPROMPTFORGE_CK_ROOT=$CK_ROOT \
  -DLLAMA_BUILD_SERVER=ON \
  -DLLAMA_BUILD_WEBUI=OFF
cmake --build build-promptforge --target llama-server -j$(nproc)

LLAMA_BUILD_WEBUI=OFF is required. The embedded browser UI downloads assets from an HF bucket (llama-ui) that returns 404; the service is consumed through the OpenAI-compatible API, so the UI is not needed.

Build output: build-promptforge/bin/llama-server.

graph TD
    A[Clone ROCmFPX v2.3 + CK pinned] --> B[Configure CK, copy generated headers]
    B --> C[Configure llama-server vs ROCm 7.15 SDK]
    C --> D[Build llama-server]
    D --> E[Verify --list-devices shows gfx1151]
    E --> F[Download v3 GGUF + FFN/GDN/Output-K8 + mmproj]
    F --> G[Deploy frank-default-large]
    G --> H[Health check + chat completion]

Model Download

With the HF token to the v3 staging directory /home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3/. The v3 files live there so the v2 artifacts in /home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX/ stay intact for rollback. Download ran on Dankclaw with the background script /tmp/download-v3.sh; all five files + the patch passed SHA-256 checks against the model card before the service restart.

Service

Unit: frank-default-large.service, port 7770
Script: scripts/serve-frank-large.sh, env scripts/frank-default-large.env
The script exports the ROCm 7.15 SDK libraries in LD_LIBRARY_PATH before the system ROCm. ROCM_PATH points at the devel SDK.

Launch profile

262144 context (native max), 1 slot (user request 2026-08-19; parallel slots degraded performance), f16 KV (~16 GiB), context checkpoints (16), prompt cache. Concurrent requests queue on the single slot; the server never preempts a running generation
Native MTP draft, depth 4, device ROCm0
--temp 0.7 --top-p 0.95, reasoning on (--reasoning-format deepseek puts <think> reasoning in message.reasoning_content)
v3 runtime: Output-K8 draft proxy (top-8 shortlist, Q8 rerank) + small-row W8 prompt route, full-W8 FFN/GDN views
PromptForge env: PROMPTFORGE_MODE=m2048_fused_tail1476, GGML_CUDA_GRAPH_OPT=0, PROMPTFORGE_MTP_OUTPUT_K8=1, PROMPTFORGE_MTP_OUTPUT_K8_VALIDATE=0, PROMPTFORGE_ENABLE_SMALLM_W8=1

Required environment

export PROMPTFORGE_SIDECAR=...FFN.pfs
export PROMPTFORGE_GDN_SIDECAR=...GDN.pfs
export PROMPTFORGE_MODE=m2048_fused_tail1476
export GGML_CUDA_GRAPH_OPT=0
export PROMPTFORGE_ENABLE_SMALLM_W8=1
export PROMPTFORGE_MTP_OUTPUT_K8=1
export PROMPTFORGE_MTP_OUTPUT_K8_VALIDATE=0
export PROMPTFORGE_MTP_OUTPUT_K8_PROXY=...Output-K8.pfs
# v3 qualified profile: disable the CPU argmax draft fast path and any
# IU4/ngram lane. The Output-K8 proxy + backend sampling replace them.
unset LLAMA_MTP_CPU_ARGMAX_FASTPATH LLAMA_MTP_BACKEND_GREEDY
unset PROMPTFORGE_IU4_SIDECAR PROMPTFORGE_GDN_IU4_SIDECAR
unset PROMPTFORGE_ENABLE_NGRAM_MOD PROMPTFORGE_ENABLE_NGRAM_M65_IU4

The v3 qualified profile runs the MTP draft through the Output-K8 proxy, not the CPU argmax fast path. The old LLAMA_MTP_CPU_ARGMAX_FASTPATH=1 profile is retired for this slot. A TOP_K sampler warning may still appear in the journal once per start; the Output-K8 draft path does not depend on it and generation proceeds.

Operational Notes

Health: curl http://127.0.0.1:7770/health. Returns 503 while loading.
Model load is slow (v3 five-file payload ~36.7 GiB); do not kill the process during warmup. The page cache makes a repeat load fast right after download, but a later cold start takes several minutes.
deploy.sh dankclaw waits 20 minutes for health.
The systemd unit must not set HOST or TEMP env vars (HIP collision rule). --host/--port/--temp are CLI flags only.
v3 adds the BF16 vision tower over v2. The service reports multimodal capability; mmproj loads with the language model (CLIP on ROCm0 backend).

ngram-mod + MTP Stacking (benchmarked 2026-08-21)

This page records a research question: does ngram-mod add value when stacked under the native MTP draft? The v3 profile runs MTP alone; the benchmark below compares that against the stacked pair.

Feasibility

ngram-mod DOES stack with MTP in this CIRU fork. The v3 build compiles in both ngram-mod and draft-mtp (common/speculative.cpp). Pass --spec-type draft-mtp,ngram-mod (or ngram-mod,draft-mtp) and the fork initializes both implementations in the same process. The boot log shows:

adding speculative implementation 'ngram-mod'
initialized ngram_mod with n_match=24, size=4194304 (16.000 MB)
adding speculative implementation 'draft-mtp'

The model card's "no ngram subsystem" line refers to the CIRU PromptForge env lanes (PROMPTFORGE_*_NGRAM_*, PROMPTFORGE_ENABLE_NGRAM_MOD), not the upstream llama.cpp spec chain. Those env lanes must stay unset; the --spec-type ngram-mod chain is separate and works.

Ordering — the CLI order does not matter

The --spec-type string order is discarded. common_speculative_init collapses the requested types into a bitmask, then rebuilds the chain in a hard-coded priority: ngram types first, then draft-simple, then MTP. So draft-mtp,ngram-mod and ngram-mod,draft-mtp produce the same internal order (ngram-mod then draft-mtp) and the same runtimes. Verified: both orders decode ~25.3 t/s at 8k.

Results (decode generation t/s, 200 tokens, prose prompt)

ctx	MTP only	MTP + ngram-mod	verdict
8k	29.42	25.27	ngram hurts -4.15
16k	26.42	21.13	ngram hurts -5.29
64k	19.72	18.00	ngram hurts -1.72

At every depth, adding ngram-mod under MTP REDUCED decode throughput. The ngram draft costs its own forward pass / occupancy without adding enough accepted tokens on top of MTP. Baseline (no ngram) remains the production config (--spec-type draft-mtp only). Keep ngram-mod off.

The historical harness was moved to .AGENTTRASH/bench-scripts-20260901/ and is retained for audit only. it boots the v3 stack on a scratch port and reads authoritative print_timing lines. It requires the production service to be stopped.

12 KiB Raw Permalink Blame History