diff --git a/AGENTS.md b/AGENTS.md index 05898de..9f1da17 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -8,7 +8,7 @@ local machines. Topology (2026-08-21): | Target | Host | Runtime | Service | Model | Port | |--------|------|---------|---------|-------|------| | Dankserver | dank@dankserver | llama.cpp (upstream b10121) | frank-default-small | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU | 7770 | -| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (ROCm 7.15 SDK in the ling3-ciru venv) | frank-default-large | jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP, PromptForge DualView, 262K context / 1 slot | 52625 | +| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (ROCm 7.15 SDK in the ling3-ciru venv) | frank-default-large | jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v3, native MTP + Output-K8 proxy, PromptForge DualView, 262K context / 1 slot | 52625 | The 2026-08-17 redo retired the pipeline-parallel split and the FastFlowLM NPU endpoint. The 2026-08-18 swap retired the vLLM service on Dankclaw; the @@ -129,16 +129,19 @@ for the base recipe. The CIRU llama-server serves `jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` from `/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server`. The runtime is the ciru-ai/ROCmFPX tag `qwen3.8-activefpx-promptforge-v2.3` -(commit `103af8b41`) built against the private TheRock ROCm 7.15 SDK kept +(commit `103af8b41`) **plus the v3 output-K8 runtime patch** (commit +`af0eac6` on Dankclaw) built against the private TheRock ROCm 7.15 SDK kept in the retired ling3-ciru venv (`.venv/lib/python3.12/site-packages/_rocm_sdk_*`). The model card requires ROCm 7.15 for the advertised Strix Halo performance; the system ROCm 7.2.4 selects slower kernels. The serve script exports the 7.15 SDK libraries first in `LD_LIBRARY_PATH`. -The model files (v2 GGUF + FFN.pfs + GDN.pfs, ~34.9 GiB) live in -`/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX/`. The full build runbook is -in docs/ciru-activefpx-promptforge.md. The retired vLLM venvs +The v3 model files (GGUF + FFN/GDN .pfs + Output-K8 proxy + BF16 mmproj, +~36.7 GiB) live in the v3 staging directory +`/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3/`; the v2 artifacts remain +in `/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX/` for rollback. +The full build runbook is in docs/ciru-activefpx-promptforge.md. The retired vLLM venvs (`/home/dank/vllm-venv`, `/home/dank/vllm-vk-venv`) were moved to `/home/dank/.AGENTTRASH/` on 2026-08-18. @@ -195,7 +198,7 @@ graph TD - `sudo` on both boxes needs a tty. Remote sudo steps fail non-interactively. Use `echo '' | sudo -S `. - Never run `pkill -f` with a pattern that appears in your own command line. It kills your ssh session. Use `pkill -x `. -- Model load is slow: Dankserver loads the 30.6 GB Q6_K_P through RAM (`--no-mmap`, MoE experts on CPU) in a few minutes cold; Dankclaw CIRU loads the 34.9 GiB ActiveFPX payload in ~5-10 minutes. Health returns 503 while loading. Do not kill the process. `deploy.sh` waits up to 20 minutes on both targets (see `HEALTH_WAIT`). +- Model load is slow: Dankserver loads the 30.6 GB Q6_K_P through RAM (`--no-mmap`, MoE experts on CPU) in a few minutes cold; Dankclaw CIRU loads the v3 ~36.7 GiB payload in ~5-10 minutes. Health returns 503 while loading. Do not kill the process. `deploy.sh` waits up to 20 minutes on both targets (see `HEALTH_WAIT`). - A client abort and a server release look different in the journal. A client abort logs `W srv stop: cancel task, id_task = N`. A natural stop (EOS or limit) logs only `slot release` with `truncated = 0`. The server never preempts a running generation; when all slots are busy, new tasks defer into a queue. See the 2026-08-13 incident in [docs/deployment.md](docs/deployment.md). - Retired hardware history: the Dankserver 35B cuBLAS crash and the DFlash premature-EOS bug are historical; see [docs/dankserver-vram-growth.md](docs/dankserver-vram-growth.md) and [docs/qwen3.8-spec-matrix.md](docs/qwen3.8-spec-matrix.md). The FastFlowLM NPU endpoint is retired; see [docs/FastFlowLM NPU Inference.md](docs/FastFlowLM%20NPU%20Inference.md). diff --git a/docs/ciru-activefpx-promptforge.md b/docs/ciru-activefpx-promptforge.md index 95876a0..e005ecf 100644 --- a/docs/ciru-activefpx-promptforge.md +++ b/docs/ciru-activefpx-promptforge.md @@ -1,7 +1,7 @@ --- type: reference title: CIRU ActiveFPX PromptForge on Dankclaw -description: jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge served by the CIRU ROCmFPX llama.cpp runtime on the Strix Halo iGPU. Install runbook, ROCm 7.15 toolchain, and the frank-default-large service. +description: jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge (v3) served by the CIRU ROCmFPX llama.cpp runtime on the Strix Halo iGPU. Install runbook, ROCm 7.15 toolchain, and the frank-default-large service. tags: - llama.cpp - rocm @@ -9,40 +9,47 @@ tags: - gfx1151 - ciru - promptforge -timestamp: 2026-08-19 +timestamp: 2026-08-21 --- # CIRU ActiveFPX PromptForge on Dankclaw `frank-default-large` (port 52625) serves -`jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` through the CIRU ROCmFPX -llama.cpp runtime. This replaced the retired vLLM service that previously -occupied the slot. +`jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` **v3** through the CIRU +ROCmFPX llama.cpp runtime. This replaced the retired vLLM service that +previously occupied the slot. The v2 deployment moved to v3 on 2026-08-21. ## What This Is -The model is a **GGUF**, not a vLLM checkpoint. It is the quality-and-speed -release of Qwen3.8-27B for AMD ROCm. CIRU DualView keeps a compact ActiveFPX -GGUF for generation and load optional PromptForge companion views to -accelerate large prompts. The model carries native depth-4 multi-token -prediction (MTP). +The model is a **GGUF**, not a vLLM checkpoint. v3 is the consolidated +quality-and-speed release of Qwen3.8-27B for AMD ROCm: an FFN-equalized +compact body, 50 retained late-writer Q6 tensors, full-Q8 output +projection, full-W8 PromptForge view (FFN + Gated DeltaNet), the +small-row W8 route, a draft-only Output-K8 shortlist proxy with Q8 +reranking, native depth-4 MTP (no ngram), and the BF16 Qwen3.8 vision +tower + 5120-wide merger as a separate projector. | Artifact | Size | |----------|------| -| `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-v2.gguf` | 15.19 GiB | +| `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-v3-Q8-FFNEQ-LateQ6.gguf` | 15.44 GiB | | `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-FFN.pfs` | 15.95 GiB | | `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-GDN.pfs` | 3.75 GiB | -| Three-file payload | ~34.89 GiB | +| `Qwen3.8-27B-CIRU-ActiveFPX-PromptForge-Output-K8.pfs` | 0.67 GiB | +| `Qwen3.8-27B-mmproj-BF16.mmproj` | 0.87 GiB | +| Full v3 payload (GGUF + 3 .pfs + mmproj) | ~36.7 GiB | -The two `.pfs` companions are optional. Omit them to run the GGUF alone and -save ~19.7 GiB of memory. The GGUF remains independently usable. +The `.pfs` companions and the Output-K8 proxy are optional. Omit them to +run the GGUF alone and save ~20.6 GiB of memory. The GGUF remains +independently usable. ## Requirements - **ROCm 7.15 is required** for the advertised Strix Halo performance. The system ROCm (`/opt/rocm-7.2.4`) selects slower or fallback kernels. - The pinned runtime is `ciru-ai/ROCmFPX` tag - `qwen3.8-activefpx-promptforge-v2.3` (commit `103af8b41`). + `qwen3.8-activefpx-promptforge-v2.3` (commit `103af8b41`) **plus the + v3 runtime patch** `runtime/qwen38-v3-output-k8-runtime.patch`. On + Dankclaw the patch is commit `af0eac6` on top of the v2.3 checkout. - Composable Kernel at `fdf4bb7fcc984811cef48ce817d89aac064b984a`. - Validated GPU: AMD Radeon 8060S / gfx1151. @@ -62,9 +69,19 @@ The retired `ling3-ciru` package left a complete TheRock ROCm 7.15 SDK The compiler is amdclang 23. `rocm-sdk version` reports `7.15.0a20260728`. The binary is built against and run with these 7.15 libraries. -## Build Runbook (2026-08-18) +## Build Runbook (2026-08-18, v2.3; +v3 patch on 2026-08-21) -Performed on Dankclaw: +Step 1-4 below built v2.3 on 2026-08-18. On 2026-08-21 the v3 +Output-K8 runtime patch was applied on top and the server rebuilt: + +0. Apply the v3 runtime patch to the v2.3 checkout and rebuild: + ``` + % verify the patch hash: 8ba7d721ea380b060aeb0c7595b8b05f1582c0f7bf7a4b89e0910bc8e72465d5 + git am runtime/qwen38-v3-output-k8-runtime.patch # -> commit af0eac6 + cmake --build build-promptforge --target llama-server -j$(nproc) + ``` + The pre-built binary is kept as `build-promptforge/bin/llama-server.v2.3-bak` + for rollback. 1. Install host build tools (ninja-build was missing): `sudo apt install ninja-build`. 2. Clone the runtime and CK at their pinned revisions: @@ -112,15 +129,19 @@ graph TD B --> C[Configure llama-server vs ROCm 7.15 SDK] C --> D[Build llama-server] D --> E[Verify --list-devices shows gfx1151] - E --> F[Download v2 GGUF + FFN + GDN] - F --> G[Deploy frank-default-small] + E --> F[Download v3 GGUF + FFN/GDN/Output-K8 + mmproj] + F --> G[Deploy frank-default-large] G --> H[Health check + chat completion] ``` ## Model Download -Downloaded with the HF token to -`/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX/`. +With the HF token to the v3 staging directory +`/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX-v3/`. The v3 files live there +so the v2 artifacts in `/home/dank/models/CIRU-Qwen3.8-27B-ActiveFPX/` stay +intact for rollback. Download ran on Dankclaw with the background script +`/tmp/download-v3.sh`; all five files + the patch passed SHA-256 checks +against the model card before the service restart. ## Service @@ -138,8 +159,12 @@ Downloaded with the HF token to - Native MTP draft, depth 4, device ROCm0 - `--temp 0.7 --top-p 0.95`, reasoning on (`--reasoning-format deepseek` puts `` reasoning in `message.reasoning_content`) +- v3 runtime: Output-K8 draft proxy (top-8 shortlist, Q8 rerank) + + small-row W8 prompt route, full-W8 FFN/GDN views - PromptForge env: `PROMPTFORGE_MODE=m2048_fused_tail1476`, - `GGML_CUDA_GRAPH_OPT=0`, `LLAMA_MTP_CPU_ARGMAX_FASTPATH=1` + `GGML_CUDA_GRAPH_OPT=0`, `PROMPTFORGE_MTP_OUTPUT_K8=1`, + `PROMPTFORGE_MTP_OUTPUT_K8_VALIDATE=0`, + `PROMPTFORGE_ENABLE_SMALLM_W8=1` ### Required environment @@ -148,20 +173,34 @@ export PROMPTFORGE_SIDECAR=...FFN.pfs export PROMPTFORGE_GDN_SIDECAR=...GDN.pfs export PROMPTFORGE_MODE=m2048_fused_tail1476 export GGML_CUDA_GRAPH_OPT=0 -export LLAMA_MTP_CPU_ARGMAX_FASTPATH=1 +export PROMPTFORGE_ENABLE_SMALLM_W8=1 +export PROMPTFORGE_MTP_OUTPUT_K8=1 +export PROMPTFORGE_MTP_OUTPUT_K8_VALIDATE=0 +export PROMPTFORGE_MTP_OUTPUT_K8_PROXY=...Output-K8.pfs +# v3 qualified profile: disable the CPU argmax draft fast path and any +# IU4/ngram lane. The Output-K8 proxy + backend sampling replace them. +unset LLAMA_MTP_CPU_ARGMAX_FASTPATH LLAMA_MTP_BACKEND_GREEDY +unset PROMPTFORGE_IU4_SIDECAR PROMPTFORGE_GDN_IU4_SIDECAR +unset PROMPTFORGE_ENABLE_NGRAM_MOD PROMPTFORGE_ENABLE_NGRAM_M65_IU4 ``` -The `p_min=0` MTP profile uses the CPU argmax fast path. This avoids ROCm's -unsupported TOP_K operator. +The v3 qualified profile runs the MTP draft through the Output-K8 proxy, +not the CPU argmax fast path. The old `LLAMA_MTP_CPU_ARGMAX_FASTPATH=1` +profile is retired for this slot. A TOP_K sampler warning may still appear +in the journal once per start; the Output-K8 draft path does not depend on +it and generation proceeds. ## Operational Notes - Health: `curl http://127.0.0.1:52625/health`. Returns 503 while loading. -- Model load is slow (34.9 GiB payload); do not kill the process during - warmup. +- Model load is slow (v3 five-file payload ~36.7 GiB); do not kill the + process during warmup. The page cache makes a repeat load fast right after + download, but a later cold start takes several minutes. - `deploy.sh dankclaw` waits 20 minutes for health. - The systemd unit must not set `HOST` or `TEMP` env vars (HIP collision rule). `--host`/`--port`/`--temp` are CLI flags only. +- v3 adds the BF16 vision tower over v2. The service reports `multimodal` + capability; mmproj loads with the language model (CLIP on ROCm0 backend). ## See Also diff --git a/docs/deployment.md b/docs/deployment.md index 1a4bca2..71d0868 100644 --- a/docs/deployment.md +++ b/docs/deployment.md @@ -9,7 +9,7 @@ Two servers run LLM inference services. Each server pulls its scripts from this | Target | Host | Runtime | Service | Port | |--------|------|---------|---------|------| | Dankserver | dank@dankserver | upstream llama.cpp (Qwen3.6-35B-A3B Q6_K_P + ngram-mod, MoE on CPU) | frank-default-small | 7770 | -| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v2, native MTP + PromptForge DualView, 262K ctx / 1 slot) | frank-default-large | 52625 | +| Dankclaw | dank@dankclaw | CIRU ROCmFPX llama.cpp (jcbtc Qwen3.8-27B-CIRU-ActiveFPX-PromptForge v3, native MTP + Output-K8 proxy + PromptForge DualView, 262K ctx / 1 slot) | frank-default-large | 52625 | ## Workflow @@ -116,8 +116,8 @@ launches through `scripts/serve-frank-small.sh`, which reads tunables from ### Dankclaw — frank-default-large The active service on Dankclaw is the CIRU ROCmFPX llama.cpp runtime serving -`jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` (v2 GGUF + FFN/GDN -companions) under the alias `frank-default-large` on port 52625. It runs +`jcbtc/Qwen3.8-27B-CIRU-ActiveFPX-PromptForge` (v3 five-file payload: GGUF ++ FFN/GDN .pfs + Output-K8 proxy + BF16 mmproj) under the alias `frank-default-large` on port 52625. It runs `/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server` (ROCm 7.15 SDK in the retired ling3-ciru venv) with native depth-4 MTP (`--spec-type draft-mtp`), f16 KV for both K and V, full 262144 context diff --git a/scripts/deploy.sh b/scripts/deploy.sh index c6008d1..6e8ae69 100755 --- a/scripts/deploy.sh +++ b/scripts/deploy.sh @@ -42,8 +42,9 @@ REMOTE_DIR[dankclaw]="/home/dank/vllmScripts" SERVICE[dankclaw]="frank-default-large" SERVICE_HEALTH[dankclaw]="http://127.0.0.1:52625/health" DROP_CACHES[dankclaw]=0 -# CIRU llama.cpp (ROCm 7.15) loads the 34.9 GiB ActiveFPX payload + MTP -# draft and initializes PromptForge views; allow 20 minutes. +# CIRU llama.cpp (ROCm 7.15) loads the v3 five-file payload (GGUF + FFN + +# GDN + Output-K8 .pfs + mmproj, ~36.7 GiB) + MTP draft/Output-K8 proxy and +# initializes PromptForge views; allow 20 minutes. HEALTH_WAIT[dankclaw]=600 TARGET="${1:-}"