dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: document frank-default-large v3 upgrade (Output-K8 runtime, full-W8 views, BF16 vision); update AGENTS/deploy payload sizes

Browse Source
This commit is contained in:
Frank committed 2026-08-21 20:18:57 -07:00
1 parent 132e1478cf
commit 8652116cec
4 files changed
+81 -38
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

AGENTS.md
+9 -6
View File
@@ -8,7 +8,7 @@ local machines. Topology (2026-08-21):
| Target | Host | Runtime | Service | Model | Port | | Target | Host | Runtime | Service | Model | Port |
|--------|------|---------|---------|-------|------| |--------|------|---------|---------|-------|------|
| Dankserver | dank@dankserver | llama.cpp (upstream b10121) | frank-default-small | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU | 7770 | | Dankserver | dank@dankserver | llama.cpp (upstream b10121) | frank-default-small | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU | 7770 |
| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (ROCm 7.15 SDK in the ling3-ciru venv) | frank-default-large | jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP, PromptForge DualView, 262K context / 1 slot | 52625 | | Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (ROCm 7.15 SDK in the ling3-ciru venv) | frank-default-large | jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v3, native MTP + Output-K8 proxy, PromptForge DualView, 262K context / 1 slot | 52625 |
The 2026-08-17 redo retired the pipeline-parallel split and the FastFlowLM The 2026-08-17 redo retired the pipeline-parallel split and the FastFlowLM
NPU endpoint. The 2026-08-18 swap retired the vLLM service on Dankclaw; the NPU endpoint. The 2026-08-18 swap retired the vLLM service on Dankclaw; the
@@ -129,16 +129,19 @@ for the base recipe.
The CIRU llama-server serves `jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` The CIRU llama-server serves `jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge`
from `/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server`. from `/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server`.
The runtime is the ciru-ai/ROCmFPX tag `qwen3.8-activefpx-promptforge-v2.3` The runtime is the ciru-ai/ROCmFPX tag `qwen3.8-activefpx-promptforge-v2.3`
(commit `103af8b41`) built against the private TheRock ROCm 7.15 SDK kept (commit `103af8b41`) **plus the v3 output-K8 runtime patch** (commit
`af0eac6` on Dankclaw) built against the private TheRock ROCm 7.15 SDK kept
in the retired ling3-ciru venv in the retired ling3-ciru venv
(`.venv/lib/python3.12/site-packages/_rocm_sdk_*`). The model card requires (`.venv/lib/python3.12/site-packages/_rocm_sdk_*`). The model card requires
ROCm 7.15 for the advertised Strix Halo performance; the system ROCm 7.2.4 ROCm 7.15 for the advertised Strix Halo performance; the system ROCm 7.2.4
selects slower kernels. The serve script exports the 7.15 SDK libraries selects slower kernels. The serve script exports the 7.15 SDK libraries
first in `LD_LIBRARY_PATH`. first in `LD_LIBRARY_PATH`.
The model files (v2 GGUF + FFN.pfs + GDN.pfs, ~34.9 GiB) live in The v3 model files (GGUF + FFN/GDN .pfs + Output-K8 proxy + BF16 mmproj,
`/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX/`. The full build runbook is ~36.7 GiB) live in the v3 staging directory
in docs/ciru-activefpx-promptforge.md. The retired vLLM venvs `/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3/`; the v2 artifacts remain
in `/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX/` for rollback.
The full build runbook is in docs/ciru-activefpx-promptforge.md. The retired vLLM venvs
(`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to (`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to
`/home/dank/.AGENTTRASH/` on 2026-08-18. `/home/dank/.AGENTTRASH/` on 2026-08-18.
@@ -195,7 +198,7 @@ graph TD
- `sudo` on both boxes needs a tty. Remote sudo steps fail non-interactively. Use `echo '<password>' | sudo -S <command>`. - `sudo` on both boxes needs a tty. Remote sudo steps fail non-interactively. Use `echo '<password>' | sudo -S <command>`.
- Never run `pkill -f` with a pattern that appears in your own command line. It kills your ssh session. Use `pkill -x <exact-name>`. - Never run `pkill -f` with a pattern that appears in your own command line. It kills your ssh session. Use `pkill -x <exact-name>`.
- Model load is slow: Dankserver loads the 30.6 GB Q6_K_P through RAM (`--no-mmap`, MoE experts on CPU) in a few minutes cold; Dankclaw CIRU loads the 34.9 GiB ActiveFPX payload in ~5-10 minutes. Health returns 503 while loading. Do not kill the process. `deploy.sh` waits up to 20 minutes on both targets (see `HEALTH_WAIT`). - Model load is slow: Dankserver loads the 30.6 GB Q6_K_P through RAM (`--no-mmap`, MoE experts on CPU) in a few minutes cold; Dankclaw CIRU loads the v3 ~36.7 GiB payload in ~5-10 minutes. Health returns 503 while loading. Do not kill the process. `deploy.sh` waits up to 20 minutes on both targets (see `HEALTH_WAIT`).
- A client abort and a server release look different in the journal. A client abort logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in [docs/deployment.md](docs/deployment.md). - A client abort and a server release look different in the journal. A client abort logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in [docs/deployment.md](docs/deployment.md).
- Retired hardware history: the Dankserver 35B cuBLAS crash and the DFlash premature-EOS bug are historical; see [docs/dankserver-vram-growth.md](docs/dankserver-vram-growth.md) and [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md). The FastFlowLM NPU endpoint is retired; see [docs/FastFlowLM NPU Inference.md](docs/FastFlowLM%20NPU%20Inference.md). - Retired hardware history: the Dankserver 35B cuBLAS crash and the DFlash premature-EOS bug are historical; see [docs/dankserver-vram-growth.md](docs/dankserver-vram-growth.md) and [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md). The FastFlowLM NPU endpoint is retired; see [docs/FastFlowLM NPU Inference.md](docs/FastFlowLM%20NPU%20Inference.md).
docs/ciru-activefpx-promptforge.md
+66 -27
View File
@@ -1,7 +1,7 @@
--- ---
type: reference type: reference
title: CIRU ActiveFPX PromptForge on Dankclaw title: CIRU ActiveFPX PromptForge on Dankclaw
description: jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge served by the CIRU ROCmFPX llama.cpp runtime on the Strix Halo iGPU. Install runbook, ROCm 7.15 toolchain, and the frank-default-large service. description: jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge (v3) served by the CIRU ROCmFPX llama.cpp runtime on the Strix Halo iGPU. Install runbook, ROCm 7.15 toolchain, and the frank-default-large service.
tags: tags:
- llama.cpp - llama.cpp
- rocm - rocm
@@ -9,40 +9,47 @@ tags:
- gfx1151 - gfx1151
- ciru - ciru
- promptforge - promptforge
timestamp: 2026-08-19 timestamp: 2026-08-21
--- ---
# CIRU ActiveFPX PromptForge on Dankclaw # CIRU ActiveFPX PromptForge on Dankclaw
`frank-default-large` (port 52625) serves `frank-default-large` (port 52625) serves
`jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` through the CIRU ROCmFPX `jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` **v3** through the CIRU
llama.cpp runtime. This replaced the retired vLLM service that previously ROCmFPX llama.cpp runtime. This replaced the retired vLLM service that
occupied the slot. previously occupied the slot. The v2 deployment moved to v3 on 2026-08-21.
## What This Is ## What This Is
The model is a **GGUF**, not a vLLM checkpoint. It is the quality-and-speed The model is a **GGUF**, not a vLLM checkpoint. v3 is the consolidated
release of Qwen3.8-27B for AMD ROCm. CIRU DualView keeps a compact ActiveFPX quality-and-speed release of Qwen3.8-27B for AMD ROCm: an FFN-equalized
GGUF for generation and load optional PromptForge companion views to compact body, 50 retained late-writer Q6 tensors, full-Q8 output
accelerate large prompts. The model carries native depth-4 multi-token projection, full-W8 PromptForge view (FFN + Gated DeltaNet), the
prediction (MTP). small-row W8 route, a draft-only Output-K8 shortlist proxy with Q8
reranking, native depth-4 MTP (no ngram), and the BF16 Qwen3.8 vision
tower + 5120-wide merger as a separate projector.
| Artifact | Size | | Artifact | Size |
|----------|------| |----------|------|
| `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-v2.gguf` | 15.19 GiB | | `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-v3-Q8-FFNEQ-LateQ6.gguf` | 15.44 GiB |
| `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-FFN.pfs` | 15.95 GiB | | `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-FFN.pfs` | 15.95 GiB |
| `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-GDN.pfs` | 3.75 GiB | | `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-GDN.pfs` | 3.75 GiB |
| Three-file payload | ~34.89 GiB | | `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-Output-K8.pfs` | 0.67 GiB |
| `Qwen3.8-27B-mmproj-BF16.mmproj` | 0.87 GiB |
| Full v3 payload (GGUF + 3 .pfs + mmproj) | ~36.7 GiB |
The two `.pfs` companions are optional. Omit them to run the GGUF alone and The `.pfs` companions and the Output-K8 proxy are optional. Omit them to
save ~19.7 GiB of memory. The GGUF remains independently usable. run the GGUF alone and save ~20.6 GiB of memory. The GGUF remains
independently usable.
## Requirements ## Requirements
- **ROCm 7.15 is required** for the advertised Strix Halo performance. - **ROCm 7.15 is required** for the advertised Strix Halo performance.
The system ROCm (`/opt/rocm-7.2.4`) selects slower or fallback kernels. The system ROCm (`/opt/rocm-7.2.4`) selects slower or fallback kernels.
- The pinned runtime is `ciru-ai/ROCmFPX` tag - The pinned runtime is `ciru-ai/ROCmFPX` tag
`qwen3.8-activefpx-promptforge-v2.3` (commit `103af8b41`). `qwen3.8-activefpx-promptforge-v2.3` (commit `103af8b41`) **plus the
v3 runtime patch** `runtime/qwen38-v3-output-k8-runtime.patch`. On
Dankclaw the patch is commit `af0eac6` on top of the v2.3 checkout.
- Composable Kernel at `fdf4bb7fcc984811cef48ce817d89aac064b984a`. - Composable Kernel at `fdf4bb7fcc984811cef48ce817d89aac064b984a`.
- Validated GPU: AMD Radeon 8060S / gfx1151. - Validated GPU: AMD Radeon 8060S / gfx1151.
@@ -62,9 +69,19 @@ The retired `ling3-ciru` package left a complete TheRock ROCm 7.15 SDK
The compiler is amdclang 23. `rocm-sdk version` reports `7.15.0a20260728`. The compiler is amdclang 23. `rocm-sdk version` reports `7.15.0a20260728`.
The binary is built against and run with these 7.15 libraries. The binary is built against and run with these 7.15 libraries.
## Build Runbook (2026-08-18) ## Build Runbook (2026-08-18, v2.3; +v3 patch on 2026-08-21)
Performed on Dankclaw: Step 1-4 below built v2.3 on 2026-08-18. On 2026-08-21 the v3
Output-K8 runtime patch was applied on top and the server rebuilt:
0. Apply the v3 runtime patch to the v2.3 checkout and rebuild:
```
% verify the patch hash: 8ba7d721ea380b060aeb0c7595b8b05f1582c0f7bf7a4b89e0910bc8e72465d5
git am runtime/qwen38-v3-output-k8-runtime.patch # -> commit af0eac6
cmake --build build-promptforge --target llama-server -j$(nproc)
```
The pre-built binary is kept as `build-promptforge/bin/llama-server.v2.3-bak`
for rollback.
1. Install host build tools (ninja-build was missing): `sudo apt install ninja-build`. 1. Install host build tools (ninja-build was missing): `sudo apt install ninja-build`.
2. Clone the runtime and CK at their pinned revisions: 2. Clone the runtime and CK at their pinned revisions:
@@ -112,15 +129,19 @@ graph TD
B --> C[Configure llama-server vs ROCm 7.15 SDK] B --> C[Configure llama-server vs ROCm 7.15 SDK]
C --> D[Build llama-server] C --> D[Build llama-server]
D --> E[Verify --list-devices shows gfx1151] D --> E[Verify --list-devices shows gfx1151]
E --> F[Download v2 GGUF + FFN + GDN] E --> F[Download v3 GGUF + FFN/GDN/Output-K8 + mmproj]
F --> G[Deploy frank-default-small] F --> G[Deploy frank-default-large]
G --> H[Health check + chat completion] G --> H[Health check + chat completion]
``` ```
## Model Download ## Model Download
Downloaded with the HF token to With the HF token to the v3 staging directory
`/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX/`. `/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3/`. The v3 files live there
so the v2 artifacts in `/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX/` stay
intact for rollback. Download ran on Dankclaw with the background script
`/tmp/download-v3.sh`; all five files + the patch passed SHA-256 checks
against the model card before the service restart.
## Service ## Service
@@ -138,8 +159,12 @@ Downloaded with the HF token to
- Native MTP draft, depth 4, device ROCm0 - Native MTP draft, depth 4, device ROCm0
- `--temp 0.7 --top-p 0.95`, reasoning on (`--reasoning-format deepseek` - `--temp 0.7 --top-p 0.95`, reasoning on (`--reasoning-format deepseek`
puts `<think>` reasoning in `message.reasoning_content`) puts `<think>` reasoning in `message.reasoning_content`)
- v3 runtime: Output-K8 draft proxy (top-8 shortlist, Q8 rerank) +
small-row W8 prompt route, full-W8 FFN/GDN views
- PromptForge env: `PROMPTFORGE_MODE=m2048_fused_tail1476`, - PromptForge env: `PROMPTFORGE_MODE=m2048_fused_tail1476`,
`GGML_CUDA_GRAPH_OPT=0`, `LLAMA_MTP_CPU_ARGMAX_FASTPATH=1` `GGML_CUDA_GRAPH_OPT=0`, `PROMPTFORGE_MTP_OUTPUT_K8=1`,
`PROMPTFORGE_MTP_OUTPUT_K8_VALIDATE=0`,
`PROMPTFORGE_ENABLE_SMALLM_W8=1`
### Required environment ### Required environment
@@ -148,20 +173,34 @@ export PROMPTFORGE_SIDECAR=...FFN.pfs
export PROMPTFORGE_GDN_SIDECAR=...GDN.pfs export PROMPTFORGE_GDN_SIDECAR=...GDN.pfs
export PROMPTFORGE_MODE=m2048_fused_tail1476 export PROMPTFORGE_MODE=m2048_fused_tail1476
export GGML_CUDA_GRAPH_OPT=0 export GGML_CUDA_GRAPH_OPT=0
export LLAMA_MTP_CPU_ARGMAX_FASTPATH=1 export PROMPTFORGE_ENABLE_SMALLM_W8=1
export PROMPTFORGE_MTP_OUTPUT_K8=1
export PROMPTFORGE_MTP_OUTPUT_K8_VALIDATE=0
export PROMPTFORGE_MTP_OUTPUT_K8_PROXY=...Output-K8.pfs
# v3 qualified profile: disable the CPU argmax draft fast path and any
# IU4/ngram lane. The Output-K8 proxy + backend sampling replace them.
unset LLAMA_MTP_CPU_ARGMAX_FASTPATH LLAMA_MTP_BACKEND_GREEDY
unset PROMPTFORGE_IU4_SIDECAR PROMPTFORGE_GDN_IU4_SIDECAR
unset PROMPTFORGE_ENABLE_NGRAM_MOD PROMPTFORGE_ENABLE_NGRAM_M65_IU4
``` ```
The `p_min=0` MTP profile uses the CPU argmax fast path. This avoids ROCm's The v3 qualified profile runs the MTP draft through the Output-K8 proxy,
unsupported TOP_K operator. not the CPU argmax fast path. The old `LLAMA_MTP_CPU_ARGMAX_FASTPATH=1`
profile is retired for this slot. A TOP_K sampler warning may still appear
in the journal once per start; the Output-K8 draft path does not depend on
it and generation proceeds.
## Operational Notes ## Operational Notes
- Health: `curl http://127.0.0.1:52625/health`. Returns 503 while loading. - Health: `curl http://127.0.0.1:52625/health`. Returns 503 while loading.
- Model load is slow (34.9 GiB payload); do not kill the process during - Model load is slow (v3 five-file payload ~36.7 GiB); do not kill the
warmup. process during warmup. The page cache makes a repeat load fast right after
download, but a later cold start takes several minutes.
- `deploy.sh dankclaw` waits 20 minutes for health. - `deploy.sh dankclaw` waits 20 minutes for health.
- The systemd unit must not set `HOST` or `TEMP` env vars (HIP collision - The systemd unit must not set `HOST` or `TEMP` env vars (HIP collision
rule). `--host`/`--port`/`--temp` are CLI flags only. rule). `--host`/`--port`/`--temp` are CLI flags only.
- v3 adds the BF16 vision tower over v2. The service reports `multimodal`
capability; mmproj loads with the language model (CLIP on ROCm0 backend).
## See Also ## See Also
docs/deployment.md
+3 -3
View File
@@ -9,7 +9,7 @@ Two servers run LLM inference services. Each server pulls its scripts from this
| Target | Host | Runtime | Service | Port | | Target | Host | Runtime | Service | Port |
|--------|------|---------|---------|------| |--------|------|---------|---------|------|
| Dankserver | dank@dankserver | upstream llama.cpp (Qwen3.6-35B-A3B Q6_K_P + ngram-mod, MoE on CPU) | frank-default-small | 7770 | | Dankserver | dank@dankserver | upstream llama.cpp (Qwen3.6-35B-A3B Q6_K_P + ngram-mod, MoE on CPU) | frank-default-small | 7770 |
| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP + PromptForge DualView, 262K ctx / 1 slot) | frank-default-large | 52625 | | Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v3, native MTP + Output-K8 proxy + PromptForge DualView, 262K ctx / 1 slot) | frank-default-large | 52625 |
## Workflow ## Workflow
@@ -116,8 +116,8 @@ launches through `scripts/serve-frank-small.sh`, which reads tunables from
### Dankclaw — frank-default-large ### Dankclaw — frank-default-large
The active service on Dankclaw is the CIRU ROCmFPX llama.cpp runtime serving The active service on Dankclaw is the CIRU ROCmFPX llama.cpp runtime serving
`jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` (v2 GGUF + FFN/GDN `jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` (v3 five-file payload: GGUF
companions) under the alias `frank-default-large` on port 52625. It runs + FFN/GDN .pfs + Output-K8 proxy + BF16 mmproj) under the alias `frank-default-large` on port 52625. It runs
`/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server` (ROCm `/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server` (ROCm
7.15 SDK in the retired ling3-ciru venv) with native depth-4 MTP 7.15 SDK in the retired ling3-ciru venv) with native depth-4 MTP
(`--spec-type draft-mtp`), f16 KV for both K and V, full 262144 context (`--spec-type draft-mtp`), f16 KV for both K and V, full 262144 context
scripts/deploy.sh
+3 -2
View File
@@ -42,8 +42,9 @@ REMOTE_DIR[dankclaw]="/home/dank/vllmScripts"
SERVICE[dankclaw]="frank-default-large" SERVICE[dankclaw]="frank-default-large"
SERVICE_HEALTH[dankclaw]="http://127.0.0.1:52625/health" SERVICE_HEALTH[dankclaw]="http://127.0.0.1:52625/health"
DROP_CACHES[dankclaw]=0 DROP_CACHES[dankclaw]=0
# CIRU llama.cpp (ROCm 7.15) loads the 34.9 GiB ActiveFPX payload + MTP # CIRU llama.cpp (ROCm 7.15) loads the v3 five-file payload (GGUF + FFN +
# draft and initializes PromptForge views; allow 20 minutes. # GDN + Output-K8 .pfs + mmproj, ~36.7 GiB) + MTP draft/Output-K8 proxy and
# initializes PromptForge views; allow 20 minutes.
HEALTH_WAIT[dankclaw]=600 HEALTH_WAIT[dankclaw]=600
TARGET="${1:-}" TARGET="${1:-}"