From 86d5f3814256d2b2036e888e901e64f5837ca010 Mon Sep 17 00:00:00 2001 From: Frank Date: Tue, 25 Aug 2026 13:36:07 -0700 Subject: [PATCH] Restore Q6 AGENT large model --- AGENTS.md | 36 ++++++++++++++------------------- docs/LLM/deployment.md | 20 +++++++++--------- docs/LLM/index.md | 13 ++++++------ docs/LLM/kingjones-strix-mtp.md | 11 +++++----- docs/index.md | 6 +++--- scripts/frank-default-large.env | 10 ++++----- scripts/serve-frank-large.sh | 8 ++++---- tests/frank_large_config.bats | 4 ++-- 8 files changed, 51 insertions(+), 57 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index f9df80b..1b83271 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -27,7 +27,7 @@ host. | Target | Runtime | Service | Model | Port | |--------|---------|---------|-------|------| | Dankserver | llama.cpp (upstream b10121) | frank-default-small | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU | 7770 | -| Dankclaw | charlie12345/ROCmFPX llama.cpp | frank-default-large | kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT Q8_0_ROCMFPX (standalone MTP draft head, Turbo4 V cache, 262K context / 1 slot), reasoning on | 7770 | +| Dankclaw | charlie12345/ROCmFPX llama.cpp | frank-default-large | kingjones Qwen3.8 Uncensored Q6_0_ROCmFPX_AGENT (standalone MTP draft head, Turbo4 V cache, 262K context / 1 slot), reasoning on | 7770 | History: the 2026-08-17 redo retired the pipeline-parallel split and the FastFlowLM NPU endpoint. The 2026-08-18 swap retired the vLLM service on @@ -35,8 +35,9 @@ Dankclaw. On 2026-08-21 a DFlash2 Vulkan deployment briefly replaced CIRU, then was reverted the same day for speed. On 2026-08-22 the service briefly swapped to the Kairic Edge IU4 runtime, then was reverted to v3 the same day (operator decision). On 2026-08-25 the service switched to the kingjones -Qwen3.8-27B-ROCmFPX-Q8_0-AGENT model (standalone MTP draft, 262144 context, -reasoning on); the previous Kairic Edge model was retired the same day. Runbooks: +Qwen3.8-27B-Uncensored-Q6_0-ROCmFPX_AGENT model (standalone MTP draft, +262144 context, reasoning on); the previous Q8_0 AGENT model was retired after +its `set_rows` crash. Runbooks: [docs/LLM/index.md](docs/LLM/index.md). Retired scripts live in `old/` (reference only) and `.AGENTTRASH/`. @@ -182,29 +183,22 @@ When you rename a service or move a model to a new host, update ## Runtime & Troubleshooting -### kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT llama.cpp (Dankclaw, ACTIVE) +### kingjones Qwen3.8 Uncensored Q6_0 ROCmFPX AGENT llama.cpp (Dankclaw, ACTIVE) -The large service serves `kingjones/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT` -(public) from `/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT/` -with a charlie12345/ROCmFPX `llama-server` (build `build-kairic`). It runs a +The large service serves `kingjones/Qwen3.8-Uncensored-ROCmFP4-STRIX-MTP` +(public) from `/home/dank/models/kingjones-Qwen3.8-27B-STRIX-MTP/` with a +charlie12345/ROCmFPX `llama-server` (build `build-promptforge`). It runs a standalone MTP draft head on ROCm0, f16 K + TurboQuant V KV cache, a full 262144 context / 1 slot, and reasoning on (Sharp template, budget -1, `--reasoning-format none`; chat kwargs keep `preserve_thinking` + -`preserve_reasoning`). The payload (~28 GB main + 1.7 GB draft + 0.9 GB -mmproj, ~30 GB total) plus the 262k KV cache fit in the lifted 114 GiB TTM/GTT -budget (raised 2026-08-25). Env `scripts/frank-default-large.env`, serve -`scripts/serve-frank-large.sh`, runbook -[docs/LLM/kingjones-q8agent-rocmfpx.md](docs/LLM/kingjones-q8agent-rocmfpx.md). +`preserve_reasoning`). The Q6_0 main model is 24.86 GB. Env +`scripts/frank-default-large.env`, serve `scripts/serve-frank-large.sh`, and +runbook `docs/LLM/kingjones-q8agent-rocmfpx.md` define the deployment. -**Serving status (2026-08-25):** the model loads but crashes in -`ggml_compute_forward_set_rows` (NULL dispatch in `libggml-cpu.so.0`) before it -can serve. Tested on all five available ROCmFPX builds (build-kairic v81, -build-promptforge v3, build-rocmi4-w4a4 v257, build-strix-rocmfp4 v209) and the -all-CPU path; all crash. Not GPU-specific, not MTP-specific, not fixed by -`LLAMA_SET_ROWS=0`. Fix is a rebuild of charlie12345/ROCmFPX with a working -`set_rows`. The proven serving context is 131072 at `gl=45` (262144 OOMs on -device-local at `gl=999`). Config goals met: 114 GiB TTM/GTT, reasoning on, -`preserve_thinking`/`preserve_reasoning` on. +**Serving status (2026-09-01):** restored the previously deployed Q6_0 AGENT +model. The Q8_0 AGENT experiment is documented in +`docs/LLM/kingjones-q8agent-rocmfpx.md`; it was retired because it crashed in +`ggml_compute_forward_set_rows` and did not provide a working service. ### charlie12345/ROCmFPX ROCmI4 W4A4 llama.cpp (Dankclaw, prior v3, 2026-08-18 to 2026-08-25) diff --git a/docs/LLM/deployment.md b/docs/LLM/deployment.md index 895364b..28eecc6 100644 --- a/docs/LLM/deployment.md +++ b/docs/LLM/deployment.md @@ -18,7 +18,7 @@ repository to a running service. | Target | Host | Runtime | Service | Port | |--------|------|---------|---------|------| | Dankserver | dank@dankserver | upstream llama.cpp (Ornith-1.5-35B-A3B-CRACK Q6_K, native MTP draft, selected MoE experts offloaded via --override-tensor) | frank-default-small | 7770 | -| Dankclaw | dank@dankclaw | charlie12345/ROCmFPX llama.cpp (caf-eemin Qwen3.8-27B ROCmI4 Q4_0, embedded MTP, ROCmI4 W4A4, 262K ctx / 1 slot) | frank-default-large | 7770 | +| Dankclaw | dank@dankclaw | charlie12345/ROCmFPX llama.cpp (kingjones Qwen3.8 Uncensored Q6_0 ROCmFPX AGENT, standalone MTP, 262K ctx / 1 slot) | frank-default-large | 7770 | ## Workflow @@ -138,21 +138,21 @@ from `peculiar-ragdoll/Qwen-Sharp-Chat-Templates`. See [[dealign-ornith-crack]]. ### Dankclaw — frank-default-large The active service on Dankclaw uses the ROCmFPX llama.cpp runtime to serve -[kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF](https://huggingface.co/kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF) +[kingjones777/Qwen3.8-Uncensored-ROCmFP4-STRIX-MTP-GGUF](https://huggingface.co/kingjones777/Qwen3.8-Uncensored-ROCmFP4-STRIX-MTP-GGUF) under the alias `frank-default-large` on port 7770. The main model is -`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf`; the standalone MTP draft is -`mtp-Qwen3.8-27B-Q4_0.gguf`, and the BF16 vision projector is loaded alongside +`Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf`; the standalone MTP draft is +`mtp-Qwen3.8-27B-Q4_0.gguf`, and the Q8_0 vision projector is loaded alongside it. The K-side KV cache remains `f16`; the V-side cache uses `turbo4`. The -service keeps the existing 65536 context / 1-slot profile. The unit file is +service uses the 262144 context / 1-slot profile. The unit file is `scripts/frank-default-large.service`; it launches through `scripts/serve-frank-large.sh`, which reads tunables from `scripts/frank-default-large.env`, applying the Qwen-Sharp chat template. -> The kingjones ROCmFP4 STRIX MTP model served the slot from 2026-08-24; it -> was replaced by the caf-eemin ROCmI4 embedded-MTP model on 2026-08-25. The -> v3 PromptForge payload and the CIRU ROCmFPX runtime are retired on this slot. -> The intermediate DFlash2 Vulkan deployment was retired the same day (2026-08-21). -> See [[dflash2-large-vulkan]] for the archived runbook. +> The Q8_0 AGENT experiment was retired on 2026-09-01 after repeated +> `set_rows` crashes and device-local allocation failures. The former caf-eemin +> ROCmI4 embedded-MTP model and the v3 PromptForge payload are historical +> deployments. The intermediate DFlash2 Vulkan deployment was retired on +> 2026-08-21. See [[kingjones-q8agent-rocmfpx]] and [[dflash2-large-vulkan]]. ## Validation diff --git a/docs/LLM/index.md b/docs/LLM/index.md index 607a245..697e3d2 100644 --- a/docs/LLM/index.md +++ b/docs/LLM/index.md @@ -3,7 +3,7 @@ Documentation for the LLM inference fleet on two local machines (2026-09-01): upstream llama.cpp on Dankserver (frank-default-small, Ornith-1.5-35B-A3B CRACK) and ROCmFPX llama.cpp on Dankclaw -(frank-default-large, kingjones Qwen3.8-27B ROCmFPX Q8 AGENT). Both +(frank-default-large, kingjones Qwen3.8 Uncensored Q6 AGENT). Both services run the Qwen-Sharp chat template. > 2026-08-22: the former `vllmScripts` repository is merged into the @@ -28,7 +28,7 @@ ngram-mod, selected experts offloaded via override, 491520 ctx] D[gfx1151 iGPU 64 GiB unified memory] E[charlie12345/ROCmFPX build-kairic (v81) frank-default-large, port 7770] - F[Qwen3.8-27B ROCmFPX Q8_0 AGENT + F[Qwen3.8 Uncensored Q6 ROCmFPX AGENT MTP draft + Turbo4 V cache, 262K ctx / 1 slot] D --> E --> F end @@ -48,14 +48,15 @@ MTP draft + Turbo4 V cache, 262K ctx / 1 slot] - [[deployment]] — deploy and update process for both servers - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change -- [kingjones Qwen3.8-27B ROCmFPX Q8 AGENT](https://huggingface.co/kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF) — active frank-default-large model on Dankclaw; standalone MTP and Turbo4 V cache -- [[kingjones-q8agent-rocmfpx]] — the Q8_0-AGENT runbook: serving profile, download, 262144 context, reasoning + preserve_thinking, 114 GiB TTM/GTT +- [kingjones Qwen3.8 Uncensored ROCmFP4 STRIX MTP](https://huggingface.co/kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF) — active frank-default-large model on Dankclaw; Q6 AGENT main model with standalone MTP +- [[kingjones-strix-mtp]] — the active Q6 AGENT runbook: serving profile, standalone MTP, reasoning + preserve_thinking, 114 GiB TTM/GTT +- [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure - [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B embedded-MTP model on Dankclaw (retired) - [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on Dankserver (frank-default-small, active 2026-08-24) -- [[kingjones-strix-mtp]] — the retired kingjones Qwen3.8-27B ROCmFP4 STRIX MTP - model on Dankclaw (superseded 2026-08-25 by the ROCmI4 embedded-MTP model) +- [[kingjones-strix-mtp]] — the active kingjones Qwen3.8-27B ROCmFP4 STRIX MTP + model on Dankclaw (restored 2026-09-01 after the Q8_0 AGENT experiment) - [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on Dankserver (frank-default-small, active 2026-08-24) - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime diff --git a/docs/LLM/kingjones-strix-mtp.md b/docs/LLM/kingjones-strix-mtp.md index 441223e..20203af 100644 --- a/docs/LLM/kingjones-strix-mtp.md +++ b/docs/LLM/kingjones-strix-mtp.md @@ -15,13 +15,12 @@ timestamp: 2026-08-24 # kingjones Qwen3.8-27B ROCmFP4 STRIX MTP on Dankclaw -> **Retired 2026-08-25.** The kingjones ROCmFP4 STRIX MTP payload has been -> replaced by the caf-eemin Qwen3.8-27B ROCmI4 embedded-MTP model on the same -> port and iGPU, under a new runtime (charlie12345/ROCmFPX ROCmI4 W4A4). See -> [[eemin-rocmi4-mtp]]. The old kingjones GGUF + standalone MTP Q4_0 draft are -> retired on this slot. The CIRU ROCmFPX runtime/binary is also retired here. +> **Active again 2026-09-01.** The kingjones ROCmFP4 STRIX MTP payload is +> restored as `frank-default-large` after the Q8_0 AGENT experiment. The Q8_0 +> model was retired because it crashed before serving. The Q6 AGENT model, +> standalone MTP draft, Q8 projector, and PromptForge runtime are now active. -> _This page documents the retired model and its runbook for reference._ +> This page documents the active model and its runbook. `frank-default-large` (port 7770) serves `Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP` from `kingjones777` on Hugging diff --git a/docs/index.md b/docs/index.md index 8667eaf..ae451c4 100644 --- a/docs/index.md +++ b/docs/index.md @@ -97,9 +97,9 @@ The LLM inference fleet on the two hosts. Full index: - [[deployment]] — unified deploy and update process for both servers - [[frank-redo-2026-08-17]] — the 2026-08-17 topology change -- [[eemin-rocmi4-mtp]] — the caf-eemin Qwen3.8-27B ROCmI4 embedded-MTP model on Dankclaw (frank-default-large, active) -- [[kingjones-strix-mtp]] — the RETIRED kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (superseded 2026-08-25) -- [[kingjones-q8agent-rocmfpx]] — the Q8_0-AGENT runbook for frank-default-large (active): serving, download, 262144 context, reasoning + preserve_thinking, 114 GiB TTM/GTT +- [[kingjones-strix-mtp]] — the active kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (restored 2026-09-01) +- [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure +- [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B ROCmI4 embedded-MTP model on Dankclaw - [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24) - [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active) - [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on Dankserver (active) diff --git a/scripts/frank-default-large.env b/scripts/frank-default-large.env index 193b7a8..4dc9c42 100644 --- a/scripts/frank-default-large.env +++ b/scripts/frank-default-large.env @@ -1,4 +1,4 @@ -# Tunables for frank-default-large (kingjones Qwen3.8-27B ROCmFPX Q8 AGENT, +# Tunables for frank-default-large (kingjones Qwen3.8 Uncensored Q6 AGENT, # dankclaw, Strix Halo gfx1151). # # Runtime: ROCmFPX llama.cpp with TurboQuant support. Serves the kingjones @@ -10,11 +10,11 @@ # llama.cpp reads --host/--port/--temp as CLI flags, never as env vars. # ROCmFPX build with TurboQuant KV-cache support. -LLAMA_BIN=/home/dank/ciru-qwen38/ROCmFPX/build-kairic/bin/llama-server +LLAMA_BIN=/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server -# Model files from https://huggingface.co/kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF. -MODEL_DIR=/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT -MODEL_FILE=Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf +# Model files from https://huggingface.co/kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF. +MODEL_DIR=/home/dank/models/kingjones-Qwen3.8-27B-STRIX-MTP +MODEL_FILE=Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf MTP_DRAFT_FILE=$MODEL_DIR/mtp-Qwen3.8-27B-Q4_0.gguf MMPROJ_FILE=mmproj-Qwen3.8-27B-BF16.gguf diff --git a/scripts/serve-frank-large.sh b/scripts/serve-frank-large.sh index 4961e74..8ba266f 100755 --- a/scripts/serve-frank-large.sh +++ b/scripts/serve-frank-large.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash -# Serve the kingjones Qwen3.8-27B ROCmFPX Q8 AGENT model as frank-default-large. +# Serve the kingjones Qwen3.8 Uncensored Q6 AGENT model as frank-default-large. # The deployment uses a standalone MTP draft head and Turbo4 V-side KV cache. set -euo pipefail @@ -12,9 +12,9 @@ if [[ -f "$ENV_FILE" ]]; then set +a fi -: "${LLAMA_BIN:=/home/dank/ciru-qwen38/ROCmFPX/build-kairic/bin/llama-server}" -: "${MODEL_DIR:=/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT}" -: "${MODEL_FILE:=Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf}" +: "${LLAMA_BIN:=/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server}" +: "${MODEL_DIR:=/home/dank/models/kingjones-Qwen3.8-27B-STRIX-MTP}" +: "${MODEL_FILE:=Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf}" : "${MTP_DRAFT_FILE:=$MODEL_DIR/mtp-Qwen3.8-27B-Q4_0.gguf}" : "${MMPROJ_FILE:=mmproj-Qwen3.8-27B-BF16.gguf}" : "${ALIAS:=frank-default-large}" diff --git a/tests/frank_large_config.bats b/tests/frank_large_config.bats index e320d92..f0db0d6 100644 --- a/tests/frank_large_config.bats +++ b/tests/frank_large_config.bats @@ -11,14 +11,14 @@ UNIT_FILE="$REPO_ROOT/scripts/frank-default-large.service" TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja" @test "frank-default-large serves the kingjones AGENT model" { - grep -q '^MODEL_FILE=Qwen3\.8-27B-Q8_0_ROCMFPX_AGENT\.gguf$' "$ENV_FILE" + grep -q '^MODEL_FILE=Qwen3\.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT\.gguf$' "$ENV_FILE" grep -q '^MTP_DRAFT_FILE=' "$ENV_FILE" ! grep -q -- '--kairic-edge' "$SERVE_FILE" } @test "frank-default-large uses the ROCmFPX runtime" { grep -q 'ROCmFPX/.*/bin/llama-server' "$ENV_FILE" - grep -q 'build-kairic/bin/llama-server' "$ENV_FILE" + grep -q 'build-promptforge/bin/llama-server' "$ENV_FILE" } @test "frank-default-large runs standalone MTP" {