dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: document frank-default-large v3 upgrade (Output-K8 runtime, full-W8 views, BF16 vision); update AGENTS/deploy payload sizes

Browse Source
This commit is contained in:
Frank
2026-08-21 20:18:57 -07:00
parent 132e1478cf
commit 8652116cec
4 changed files with 81 additions and 38 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+9 -6
View File
@@ -8,7 +8,7 @@ local machines. Topology (2026-08-21):
| Target | Host | Runtime | Service | Model | Port |
|--------|------|---------|---------|-------|------|
| Dankserver | dank@dankserver | llama.cpp (upstream b10121) | frank-default-small | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU | 7770 |
| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (ROCm 7.15 SDK in the ling3-ciru venv) | frank-default-large | jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP, PromptForge DualView, 262K context / 1 slot | 52625 |
| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (ROCm 7.15 SDK in the ling3-ciru venv) | frank-default-large | jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v3, native MTP + Output-K8 proxy, PromptForge DualView, 262K context / 1 slot | 52625 |
The 2026-08-17 redo retired the pipeline-parallel split and the FastFlowLM
NPU endpoint. The 2026-08-18 swap retired the vLLM service on Dankclaw; the
@@ -129,16 +129,19 @@ for the base recipe.
The CIRU llama-server serves `jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge`
from `/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server`.
The runtime is the ciru-ai/ROCmFPX tag `qwen3.8-activefpx-promptforge-v2.3`
(commit `103af8b41`) built against the private TheRock ROCm 7.15 SDK kept
(commit `103af8b41`) **plus the v3 output-K8 runtime patch** (commit
`af0eac6` on Dankclaw) built against the private TheRock ROCm 7.15 SDK kept
in the retired ling3-ciru venv
(`.venv/lib/python3.12/site-packages/_rocm_sdk_*`). The model card requires
ROCm 7.15 for the advertised Strix Halo performance; the system ROCm 7.2.4
selects slower kernels. The serve script exports the 7.15 SDK libraries
first in `LD_LIBRARY_PATH`.
The model files (v2 GGUF + FFN.pfs + GDN.pfs, ~34.9 GiB) live in
`/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX/`. The full build runbook is
in docs/ciru-activefpx-promptforge.md. The retired vLLM venvs
The v3 model files (GGUF + FFN/GDN .pfs + Output-K8 proxy + BF16 mmproj,
~36.7 GiB) live in the v3 staging directory
`/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3/`; the v2 artifacts remain
in `/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX/` for rollback.
The full build runbook is in docs/ciru-activefpx-promptforge.md. The retired vLLM venvs
(`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to
`/home/dank/.AGENTTRASH/` on 2026-08-18.
@@ -195,7 +198,7 @@ graph TD
- `sudo` on both boxes needs a tty. Remote sudo steps fail non-interactively. Use `echo '<password>' | sudo -S <command>`.
- Never run `pkill -f` with a pattern that appears in your own command line. It kills your ssh session. Use `pkill -x <exact-name>`.
- Model load is slow: Dankserver loads the 30.6 GB Q6_K_P through RAM (`--no-mmap`, MoE experts on CPU) in a few minutes cold; Dankclaw CIRU loads the 34.9 GiB ActiveFPX payload in ~5-10 minutes. Health returns 503 while loading. Do not kill the process. `deploy.sh` waits up to 20 minutes on both targets (see `HEALTH_WAIT`).
- Model load is slow: Dankserver loads the 30.6 GB Q6_K_P through RAM (`--no-mmap`, MoE experts on CPU) in a few minutes cold; Dankclaw CIRU loads the v3 ~36.7 GiB payload in ~5-10 minutes. Health returns 503 while loading. Do not kill the process. `deploy.sh` waits up to 20 minutes on both targets (see `HEALTH_WAIT`).
- A client abort and a server release look different in the journal. A client abort logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in [docs/deployment.md](docs/deployment.md).
- Retired hardware history: the Dankserver 35B cuBLAS crash and the DFlash premature-EOS bug are historical; see [docs/dankserver-vram-growth.md](docs/dankserver-vram-growth.md) and [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md). The FastFlowLM NPU endpoint is retired; see [docs/FastFlowLM NPU Inference.md](docs/FastFlowLM%20NPU%20Inference.md).
docs/ciru-activefpx-promptforge.md
+66 -27
View File
@@ -1,7 +1,7 @@
---
type: reference
title: CIRU ActiveFPX PromptForge on Dankclaw
description: jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge served by the CIRU ROCmFPX llama.cpp runtime on the Strix Halo iGPU. Install runbook, ROCm 7.15 toolchain, and the frank-default-large service.
description: jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge (v3) served by the CIRU ROCmFPX llama.cpp runtime on the Strix Halo iGPU. Install runbook, ROCm 7.15 toolchain, and the frank-default-large service.
tags:
- llama.cpp
- rocm
@@ -9,40 +9,47 @@ tags:
- gfx1151
- ciru
- promptforge
timestamp: 2026-08-19
timestamp: 2026-08-21
---
# CIRU ActiveFPX PromptForge on Dankclaw
`frank-default-large` (port 52625) serves
`jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` through the CIRU ROCmFPX
llama.cpp runtime. This replaced the retired vLLM service that previously
occupied the slot.
`jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` **v3** through the CIRU
ROCmFPX llama.cpp runtime. This replaced the retired vLLM service that
previously occupied the slot. The v2 deployment moved to v3 on 2026-08-21.
## What This Is
The model is a **GGUF**, not a vLLM checkpoint. It is the quality-and-speed
release of Qwen3.8-27B for AMD ROCm. CIRU DualView keeps a compact ActiveFPX
GGUF for generation and load optional PromptForge companion views to
accelerate large prompts. The model carries native depth-4 multi-token
prediction (MTP).
The model is a **GGUF**, not a vLLM checkpoint. v3 is the consolidated
quality-and-speed release of Qwen3.8-27B for AMD ROCm: an FFN-equalized
compact body, 50 retained late-writer Q6 tensors, full-Q8 output
projection, full-W8 PromptForge view (FFN + Gated DeltaNet), the
small-row W8 route, a draft-only Output-K8 shortlist proxy with Q8
reranking, native depth-4 MTP (no ngram), and the BF16 Qwen3.8 vision
tower + 5120-wide merger as a separate projector.
| Artifact | Size |
|----------|------|
| `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-v2.gguf` | 15.19 GiB |
| `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-v3-Q8-FFNEQ-LateQ6.gguf` | 15.44 GiB |
| `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-FFN.pfs` | 15.95 GiB |
| `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-GDN.pfs` | 3.75 GiB |
| Three-file payload | ~34.89 GiB |
| `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-Output-K8.pfs` | 0.67 GiB |
| `Qwen3.8-27B-mmproj-BF16.mmproj` | 0.87 GiB |
| Full v3 payload (GGUF + 3 .pfs + mmproj) | ~36.7 GiB |
The two `.pfs` companions are optional. Omit them to run the GGUF alone and
save ~19.7 GiB of memory. The GGUF remains independently usable.
The `.pfs` companions and the Output-K8 proxy are optional. Omit them to
run the GGUF alone and save ~20.6 GiB of memory. The GGUF remains
independently usable.
## Requirements
- **ROCm 7.15 is required** for the advertised Strix Halo performance.
The system ROCm (`/opt/rocm-7.2.4`) selects slower or fallback kernels.
- The pinned runtime is `ciru-ai/ROCmFPX` tag
`qwen3.8-activefpx-promptforge-v2.3` (commit `103af8b41`).
`qwen3.8-activefpx-promptforge-v2.3` (commit `103af8b41`) **plus the
v3 runtime patch** `runtime/qwen38-v3-output-k8-runtime.patch`. On
Dankclaw the patch is commit `af0eac6` on top of the v2.3 checkout.
- Composable Kernel at `fdf4bb7fcc984811cef48ce817d89aac064b984a`.
- Validated GPU: AMD Radeon 8060S / gfx1151.
@@ -62,9 +69,19 @@ The retired `ling3-ciru` package left a complete TheRock ROCm 7.15 SDK
The compiler is amdclang 23. `rocm-sdk version` reports `7.15.0a20260728`.
The binary is built against and run with these 7.15 libraries.
## Build Runbook (2026-08-18)
## Build Runbook (2026-08-18, v2.3; +v3 patch on 2026-08-21)
Performed on Dankclaw:
Step 1-4 below built v2.3 on 2026-08-18. On 2026-08-21 the v3
Output-K8 runtime patch was applied on top and the server rebuilt:
0. Apply the v3 runtime patch to the v2.3 checkout and rebuild:
```
% verify the patch hash: 8ba7d721ea380b060aeb0c7595b8b05f1582c0f7bf7a4b89e0910bc8e72465d5
git am runtime/qwen38-v3-output-k8-runtime.patch # -> commit af0eac6
cmake --build build-promptforge --target llama-server -j$(nproc)
```
The pre-built binary is kept as `build-promptforge/bin/llama-server.v2.3-bak`
for rollback.
1. Install host build tools (ninja-build was missing): `sudo apt install ninja-build`.
2. Clone the runtime and CK at their pinned revisions:
@@ -112,15 +129,19 @@ graph TD
B --> C[Configure llama-server vs ROCm 7.15 SDK]
C --> D[Build llama-server]
D --> E[Verify --list-devices shows gfx1151]
E --> F[Download v2 GGUF + FFN + GDN]
F --> G[Deploy frank-default-small]
E --> F[Download v3 GGUF + FFN/GDN/Output-K8 + mmproj]
F --> G[Deploy frank-default-large]
G --> H[Health check + chat completion]
```
## Model Download
Downloaded with the HF token to
`/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX/`.
With the HF token to the v3 staging directory
`/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3/`. The v3 files live there
so the v2 artifacts in `/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX/` stay
intact for rollback. Download ran on Dankclaw with the background script
`/tmp/download-v3.sh`; all five files + the patch passed SHA-256 checks
against the model card before the service restart.
## Service
@@ -138,8 +159,12 @@ Downloaded with the HF token to
- Native MTP draft, depth 4, device ROCm0
- `--temp 0.7 --top-p 0.95`, reasoning on (`--reasoning-format deepseek`
puts `<think>` reasoning in `message.reasoning_content`)
- v3 runtime: Output-K8 draft proxy (top-8 shortlist, Q8 rerank) +
small-row W8 prompt route, full-W8 FFN/GDN views
- PromptForge env: `PROMPTFORGE_MODE=m2048_fused_tail1476`,
`GGML_CUDA_GRAPH_OPT=0`, `LLAMA_MTP_CPU_ARGMAX_FASTPATH=1`
`GGML_CUDA_GRAPH_OPT=0`, `PROMPTFORGE_MTP_OUTPUT_K8=1`,
`PROMPTFORGE_MTP_OUTPUT_K8_VALIDATE=0`,
`PROMPTFORGE_ENABLE_SMALLM_W8=1`
### Required environment
@@ -148,20 +173,34 @@ export PROMPTFORGE_SIDECAR=...FFN.pfs
export PROMPTFORGE_GDN_SIDECAR=...GDN.pfs
export PROMPTFORGE_MODE=m2048_fused_tail1476
export GGML_CUDA_GRAPH_OPT=0
export LLAMA_MTP_CPU_ARGMAX_FASTPATH=1
export PROMPTFORGE_ENABLE_SMALLM_W8=1
export PROMPTFORGE_MTP_OUTPUT_K8=1
export PROMPTFORGE_MTP_OUTPUT_K8_VALIDATE=0
export PROMPTFORGE_MTP_OUTPUT_K8_PROXY=...Output-K8.pfs
# v3 qualified profile: disable the CPU argmax draft fast path and any
# IU4/ngram lane. The Output-K8 proxy + backend sampling replace them.
unset LLAMA_MTP_CPU_ARGMAX_FASTPATH LLAMA_MTP_BACKEND_GREEDY
unset PROMPTFORGE_IU4_SIDECAR PROMPTFORGE_GDN_IU4_SIDECAR
unset PROMPTFORGE_ENABLE_NGRAM_MOD PROMPTFORGE_ENABLE_NGRAM_M65_IU4
```
The `p_min=0` MTP profile uses the CPU argmax fast path. This avoids ROCm's
unsupported TOP_K operator.
The v3 qualified profile runs the MTP draft through the Output-K8 proxy,
not the CPU argmax fast path. The old `LLAMA_MTP_CPU_ARGMAX_FASTPATH=1`
profile is retired for this slot. A TOP_K sampler warning may still appear
in the journal once per start; the Output-K8 draft path does not depend on
it and generation proceeds.
## Operational Notes
- Health: `curl http://127.0.0.1:52625/health`. Returns 503 while loading.
- Model load is slow (34.9 GiB payload); do not kill the process during
warmup.
- Model load is slow (v3 five-file payload ~36.7 GiB); do not kill the
process during warmup. The page cache makes a repeat load fast right after
download, but a later cold start takes several minutes.
- `deploy.sh dankclaw` waits 20 minutes for health.
- The systemd unit must not set `HOST` or `TEMP` env vars (HIP collision
rule). `--host`/`--port`/`--temp` are CLI flags only.
- v3 adds the BF16 vision tower over v2. The service reports `multimodal`
capability; mmproj loads with the language model (CLIP on ROCm0 backend).
## See Also
docs/deployment.md
+3 -3
View File
@@ -9,7 +9,7 @@ Two servers run LLM inference services. Each server pulls its scripts from this
| Target | Host | Runtime | Service | Port |
|--------|------|---------|---------|------|
| Dankserver | dank@dankserver | upstream llama.cpp (Qwen3.6-35B-A3B Q6_K_P + ngram-mod, MoE on CPU) | frank-default-small | 7770 |
| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP + PromptForge DualView, 262K ctx / 1 slot) | frank-default-large | 52625 |
| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v3, native MTP + Output-K8 proxy + PromptForge DualView, 262K ctx / 1 slot) | frank-default-large | 52625 |
## Workflow
@@ -116,8 +116,8 @@ launches through `scripts/serve-frank-small.sh`, which reads tunables from
### Dankclaw — frank-default-large
The active service on Dankclaw is the CIRU ROCmFPX llama.cpp runtime serving
`jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` (v2 GGUF + FFN/GDN
companions) under the alias `frank-default-large` on port 52625. It runs
`jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` (v3 five-file payload: GGUF
+ FFN/GDN .pfs + Output-K8 proxy + BF16 mmproj) under the alias `frank-default-large` on port 52625. It runs
`/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server` (ROCm
7.15 SDK in the retired ling3-ciru venv) with native depth-4 MTP
(`--spec-type draft-mtp`), f16 KV for both K and V, full 262144 context
scripts/deploy.sh
+3 -2
View File
@@ -42,8 +42,9 @@ REMOTE_DIR[dankclaw]="/home/dank/vllmScripts"
SERVICE[dankclaw]="frank-default-large"
SERVICE_HEALTH[dankclaw]="http://127.0.0.1:52625/health"
DROP_CACHES[dankclaw]=0
# CIRU llama.cpp (ROCm 7.15) loads the 34.9 GiB ActiveFPX payload + MTP
# draft and initializes PromptForge views; allow 20 minutes.
# CIRU llama.cpp (ROCm 7.15) loads the v3 five-file payload (GGUF + FFN +
# GDN + Output-K8 .pfs + mmproj, ~36.7 GiB) + MTP draft/Output-K8 proxy and
# initializes PromptForge views; allow 20 minutes.
HEALTH_WAIT[dankclaw]=600
TARGET="${1:-}"