Restore Q6 AGENT large model
This commit is contained in:
Frank
@@ -27,7 +27,7 @@ host.
|
||||
| Target | Runtime | Service | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | llama.cpp (upstream b10121) | frank-default-small | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU | 7770 |
|
||||
| Dankclaw | charlie12345/ROCmFPX llama.cpp | frank-default-large | kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT Q8_0_ROCMFPX (standalone MTP draft head, Turbo4 V cache, 262K context / 1 slot), reasoning on | 7770 |
|
||||
| Dankclaw | charlie12345/ROCmFPX llama.cpp | frank-default-large | kingjones Qwen3.8 Uncensored Q6_0_ROCmFPX_AGENT (standalone MTP draft head, Turbo4 V cache, 262K context / 1 slot), reasoning on | 7770 |
|
||||
|
||||
History: the 2026-08-17 redo retired the pipeline-parallel split and the
|
||||
FastFlowLM NPU endpoint. The 2026-08-18 swap retired the vLLM service on
|
||||
@@ -35,8 +35,9 @@ Dankclaw. On 2026-08-21 a DFlash2 Vulkan deployment briefly replaced CIRU,
|
||||
then was reverted the same day for speed. On 2026-08-22 the service briefly
|
||||
swapped to the Kairic Edge IU4 runtime, then was reverted to v3 the same day
|
||||
(operator decision). On 2026-08-25 the service switched to the kingjones
|
||||
Qwen3.8-27B-ROCmFPX-Q8_0-AGENT model (standalone MTP draft, 262144 context,
|
||||
reasoning on); the previous Kairic Edge model was retired the same day. Runbooks:
|
||||
Qwen3.8-27B-Uncensored-Q6_0-ROCmFPX_AGENT model (standalone MTP draft,
|
||||
262144 context, reasoning on); the previous Q8_0 AGENT model was retired after
|
||||
its `set_rows` crash. Runbooks:
|
||||
[docs/LLM/index.md](docs/LLM/index.md).
|
||||
Retired scripts live in `old/` (reference only) and `.AGENTTRASH/`.
|
||||
|
||||
@@ -182,29 +183,22 @@ When you rename a service or move a model to a new host, update
|
||||
|
||||
## Runtime & Troubleshooting
|
||||
|
||||
### kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT llama.cpp (Dankclaw, ACTIVE)
|
||||
### kingjones Qwen3.8 Uncensored Q6_0 ROCmFPX AGENT llama.cpp (Dankclaw, ACTIVE)
|
||||
|
||||
The large service serves `kingjones/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT`
|
||||
(public) from `/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT/`
|
||||
with a charlie12345/ROCmFPX `llama-server` (build `build-kairic`). It runs a
|
||||
The large service serves `kingjones/Qwen3.8-Uncensored-ROCmFP4-STRIX-MTP`
|
||||
(public) from `/home/dank/models/kingjones-Qwen3.8-27B-STRIX-MTP/` with a
|
||||
charlie12345/ROCmFPX `llama-server` (build `build-promptforge`). It runs a
|
||||
standalone MTP draft head on ROCm0, f16 K + TurboQuant V KV cache, a full
|
||||
262144 context / 1 slot, and reasoning on (Sharp template, budget -1,
|
||||
`--reasoning-format none`; chat kwargs keep `preserve_thinking` +
|
||||
`preserve_reasoning`). The payload (~28 GB main + 1.7 GB draft + 0.9 GB
|
||||
mmproj, ~30 GB total) plus the 262k KV cache fit in the lifted 114 GiB TTM/GTT
|
||||
budget (raised 2026-08-25). Env `scripts/frank-default-large.env`, serve
|
||||
`scripts/serve-frank-large.sh`, runbook
|
||||
[docs/LLM/kingjones-q8agent-rocmfpx.md](docs/LLM/kingjones-q8agent-rocmfpx.md).
|
||||
`preserve_reasoning`). The Q6_0 main model is 24.86 GB. Env
|
||||
`scripts/frank-default-large.env`, serve `scripts/serve-frank-large.sh`, and
|
||||
runbook `docs/LLM/kingjones-q8agent-rocmfpx.md` define the deployment.
|
||||
|
||||
**Serving status (2026-08-25):** the model loads but crashes in
|
||||
`ggml_compute_forward_set_rows` (NULL dispatch in `libggml-cpu.so.0`) before it
|
||||
can serve. Tested on all five available ROCmFPX builds (build-kairic v81,
|
||||
build-promptforge v3, build-rocmi4-w4a4 v257, build-strix-rocmfp4 v209) and the
|
||||
all-CPU path; all crash. Not GPU-specific, not MTP-specific, not fixed by
|
||||
`LLAMA_SET_ROWS=0`. Fix is a rebuild of charlie12345/ROCmFPX with a working
|
||||
`set_rows`. The proven serving context is 131072 at `gl=45` (262144 OOMs on
|
||||
device-local at `gl=999`). Config goals met: 114 GiB TTM/GTT, reasoning on,
|
||||
`preserve_thinking`/`preserve_reasoning` on.
|
||||
**Serving status (2026-09-01):** restored the previously deployed Q6_0 AGENT
|
||||
model. The Q8_0 AGENT experiment is documented in
|
||||
`docs/LLM/kingjones-q8agent-rocmfpx.md`; it was retired because it crashed in
|
||||
`ggml_compute_forward_set_rows` and did not provide a working service.
|
||||
|
||||
### charlie12345/ROCmFPX ROCmI4 W4A4 llama.cpp (Dankclaw, prior v3, 2026-08-18 to 2026-08-25)
|
||||
|
||||
|
||||
+10
-10
@@ -18,7 +18,7 @@ repository to a running service.
|
||||
| Target | Host | Runtime | Service | Port |
|
||||
|--------|------|---------|---------|------|
|
||||
| Dankserver | dank@dankserver | upstream llama.cpp (Ornith-1.5-35B-A3B-CRACK Q6_K, native MTP draft, selected MoE experts offloaded via --override-tensor) | frank-default-small | 7770 |
|
||||
| Dankclaw | dank@dankclaw | charlie12345/ROCmFPX llama.cpp (caf-eemin Qwen3.8-27B ROCmI4 Q4_0, embedded MTP, ROCmI4 W4A4, 262K ctx / 1 slot) | frank-default-large | 7770 |
|
||||
| Dankclaw | dank@dankclaw | charlie12345/ROCmFPX llama.cpp (kingjones Qwen3.8 Uncensored Q6_0 ROCmFPX AGENT, standalone MTP, 262K ctx / 1 slot) | frank-default-large | 7770 |
|
||||
|
||||
## Workflow
|
||||
|
||||
@@ -138,21 +138,21 @@ from `peculiar-ragdoll/Qwen-Sharp-Chat-Templates`. See [[dealign-ornith-crack]].
|
||||
### Dankclaw — frank-default-large
|
||||
|
||||
The active service on Dankclaw uses the ROCmFPX llama.cpp runtime to serve
|
||||
[kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF](https://huggingface.co/kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF)
|
||||
[kingjones777/Qwen3.8-Uncensored-ROCmFP4-STRIX-MTP-GGUF](https://huggingface.co/kingjones777/Qwen3.8-Uncensored-ROCmFP4-STRIX-MTP-GGUF)
|
||||
under the alias `frank-default-large` on port 7770. The main model is
|
||||
`Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf`; the standalone MTP draft is
|
||||
`mtp-Qwen3.8-27B-Q4_0.gguf`, and the BF16 vision projector is loaded alongside
|
||||
`Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf`; the standalone MTP draft is
|
||||
`mtp-Qwen3.8-27B-Q4_0.gguf`, and the Q8_0 vision projector is loaded alongside
|
||||
it. The K-side KV cache remains `f16`; the V-side cache uses `turbo4`. The
|
||||
service keeps the existing 65536 context / 1-slot profile. The unit file is
|
||||
service uses the 262144 context / 1-slot profile. The unit file is
|
||||
`scripts/frank-default-large.service`; it launches through
|
||||
`scripts/serve-frank-large.sh`, which reads tunables from
|
||||
`scripts/frank-default-large.env`, applying the Qwen-Sharp chat template.
|
||||
|
||||
> The kingjones ROCmFP4 STRIX MTP model served the slot from 2026-08-24; it
|
||||
> was replaced by the caf-eemin ROCmI4 embedded-MTP model on 2026-08-25. The
|
||||
> v3 PromptForge payload and the CIRU ROCmFPX runtime are retired on this slot.
|
||||
> The intermediate DFlash2 Vulkan deployment was retired the same day (2026-08-21).
|
||||
> See [[dflash2-large-vulkan]] for the archived runbook.
|
||||
> The Q8_0 AGENT experiment was retired on 2026-09-01 after repeated
|
||||
> `set_rows` crashes and device-local allocation failures. The former caf-eemin
|
||||
> ROCmI4 embedded-MTP model and the v3 PromptForge payload are historical
|
||||
> deployments. The intermediate DFlash2 Vulkan deployment was retired on
|
||||
> 2026-08-21. See [[kingjones-q8agent-rocmfpx]] and [[dflash2-large-vulkan]].
|
||||
|
||||
## Validation
|
||||
|
||||
|
||||
+7
-6
@@ -3,7 +3,7 @@
|
||||
Documentation for the LLM inference fleet on two local machines
|
||||
(2026-09-01): upstream llama.cpp on Dankserver (frank-default-small,
|
||||
Ornith-1.5-35B-A3B CRACK) and ROCmFPX llama.cpp on Dankclaw
|
||||
(frank-default-large, kingjones Qwen3.8-27B ROCmFPX Q8 AGENT). Both
|
||||
(frank-default-large, kingjones Qwen3.8 Uncensored Q6 AGENT). Both
|
||||
services run the Qwen-Sharp chat template.
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
@@ -28,7 +28,7 @@ ngram-mod, selected experts offloaded via override, 491520 ctx]
|
||||
D[gfx1151 iGPU 64 GiB unified memory]
|
||||
E[charlie12345/ROCmFPX build-kairic (v81)
|
||||
frank-default-large, port 7770]
|
||||
F[Qwen3.8-27B ROCmFPX Q8_0 AGENT
|
||||
F[Qwen3.8 Uncensored Q6 ROCmFPX AGENT
|
||||
MTP draft + Turbo4 V cache, 262K ctx / 1 slot]
|
||||
D --> E --> F
|
||||
end
|
||||
@@ -48,14 +48,15 @@ MTP draft + Turbo4 V cache, 262K ctx / 1 slot]
|
||||
|
||||
- [[deployment]] — deploy and update process for both servers
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [kingjones Qwen3.8-27B ROCmFPX Q8 AGENT](https://huggingface.co/kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF) — active frank-default-large model on Dankclaw; standalone MTP and Turbo4 V cache
|
||||
- [[kingjones-q8agent-rocmfpx]] — the Q8_0-AGENT runbook: serving profile, download, 262144 context, reasoning + preserve_thinking, 114 GiB TTM/GTT
|
||||
- [kingjones Qwen3.8 Uncensored ROCmFP4 STRIX MTP](https://huggingface.co/kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF) — active frank-default-large model on Dankclaw; Q6 AGENT main model with standalone MTP
|
||||
- [[kingjones-strix-mtp]] — the active Q6 AGENT runbook: serving profile, standalone MTP, reasoning + preserve_thinking, 114 GiB TTM/GTT
|
||||
- [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure
|
||||
- [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B embedded-MTP model
|
||||
on Dankclaw (retired)
|
||||
- [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on
|
||||
Dankserver (frank-default-small, active 2026-08-24)
|
||||
- [[kingjones-strix-mtp]] — the retired kingjones Qwen3.8-27B ROCmFP4 STRIX MTP
|
||||
model on Dankclaw (superseded 2026-08-25 by the ROCmI4 embedded-MTP model)
|
||||
- [[kingjones-strix-mtp]] — the active kingjones Qwen3.8-27B ROCmFP4 STRIX MTP
|
||||
model on Dankclaw (restored 2026-09-01 after the Q8_0 AGENT experiment)
|
||||
- [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on
|
||||
Dankserver (frank-default-small, active 2026-08-24)
|
||||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime
|
||||
|
||||
@@ -15,13 +15,12 @@ timestamp: 2026-08-24
|
||||
|
||||
# kingjones Qwen3.8-27B ROCmFP4 STRIX MTP on Dankclaw
|
||||
|
||||
> **Retired 2026-08-25.** The kingjones ROCmFP4 STRIX MTP payload has been
|
||||
> replaced by the caf-eemin Qwen3.8-27B ROCmI4 embedded-MTP model on the same
|
||||
> port and iGPU, under a new runtime (charlie12345/ROCmFPX ROCmI4 W4A4). See
|
||||
> [[eemin-rocmi4-mtp]]. The old kingjones GGUF + standalone MTP Q4_0 draft are
|
||||
> retired on this slot. The CIRU ROCmFPX runtime/binary is also retired here.
|
||||
> **Active again 2026-09-01.** The kingjones ROCmFP4 STRIX MTP payload is
|
||||
> restored as `frank-default-large` after the Q8_0 AGENT experiment. The Q8_0
|
||||
> model was retired because it crashed before serving. The Q6 AGENT model,
|
||||
> standalone MTP draft, Q8 projector, and PromptForge runtime are now active.
|
||||
|
||||
> _This page documents the retired model and its runbook for reference._
|
||||
> This page documents the active model and its runbook.
|
||||
|
||||
`frank-default-large` (port 7770) serves
|
||||
`Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP` from `kingjones777` on Hugging
|
||||
|
||||
+3
-3
@@ -97,9 +97,9 @@ The LLM inference fleet on the two hosts. Full index:
|
||||
|
||||
- [[deployment]] — unified deploy and update process for both servers
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [[eemin-rocmi4-mtp]] — the caf-eemin Qwen3.8-27B ROCmI4 embedded-MTP model on Dankclaw (frank-default-large, active)
|
||||
- [[kingjones-strix-mtp]] — the RETIRED kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (superseded 2026-08-25)
|
||||
- [[kingjones-q8agent-rocmfpx]] — the Q8_0-AGENT runbook for frank-default-large (active): serving, download, 262144 context, reasoning + preserve_thinking, 114 GiB TTM/GTT
|
||||
- [[kingjones-strix-mtp]] — the active kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (restored 2026-09-01)
|
||||
- [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure
|
||||
- [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B ROCmI4 embedded-MTP model on Dankclaw
|
||||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24)
|
||||
- [[kingjones-strix-mtp]] — the kingjones Qwen3.8-27B ROCmFP4 STRIX MTP model on Dankclaw (active)
|
||||
- [[dealign-ornith-crack]] — the dealign Ornith-1.5-35B-A3B CRACK model on Dankserver (active)
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Tunables for frank-default-large (kingjones Qwen3.8-27B ROCmFPX Q8 AGENT,
|
||||
# Tunables for frank-default-large (kingjones Qwen3.8 Uncensored Q6 AGENT,
|
||||
# dankclaw, Strix Halo gfx1151).
|
||||
#
|
||||
# Runtime: ROCmFPX llama.cpp with TurboQuant support. Serves the kingjones
|
||||
@@ -10,11 +10,11 @@
|
||||
# llama.cpp reads --host/--port/--temp as CLI flags, never as env vars.
|
||||
|
||||
# ROCmFPX build with TurboQuant KV-cache support.
|
||||
LLAMA_BIN=/home/dank/ciru-qwen38/ROCmFPX/build-kairic/bin/llama-server
|
||||
LLAMA_BIN=/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server
|
||||
|
||||
# Model files from https://huggingface.co/kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF.
|
||||
MODEL_DIR=/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT
|
||||
MODEL_FILE=Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf
|
||||
# Model files from https://huggingface.co/kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF.
|
||||
MODEL_DIR=/home/dank/models/kingjones-Qwen3.8-27B-STRIX-MTP
|
||||
MODEL_FILE=Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf
|
||||
MTP_DRAFT_FILE=$MODEL_DIR/mtp-Qwen3.8-27B-Q4_0.gguf
|
||||
MMPROJ_FILE=mmproj-Qwen3.8-27B-BF16.gguf
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
#!/usr/bin/env bash
|
||||
# Serve the kingjones Qwen3.8-27B ROCmFPX Q8 AGENT model as frank-default-large.
|
||||
# Serve the kingjones Qwen3.8 Uncensored Q6 AGENT model as frank-default-large.
|
||||
# The deployment uses a standalone MTP draft head and Turbo4 V-side KV cache.
|
||||
set -euo pipefail
|
||||
|
||||
@@ -12,9 +12,9 @@ if [[ -f "$ENV_FILE" ]]; then
|
||||
set +a
|
||||
fi
|
||||
|
||||
: "${LLAMA_BIN:=/home/dank/ciru-qwen38/ROCmFPX/build-kairic/bin/llama-server}"
|
||||
: "${MODEL_DIR:=/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT}"
|
||||
: "${MODEL_FILE:=Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf}"
|
||||
: "${LLAMA_BIN:=/home/dank/ciru-qwen38/ROCmFPX/build-promptforge/bin/llama-server}"
|
||||
: "${MODEL_DIR:=/home/dank/models/kingjones-Qwen3.8-27B-STRIX-MTP}"
|
||||
: "${MODEL_FILE:=Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf}"
|
||||
: "${MTP_DRAFT_FILE:=$MODEL_DIR/mtp-Qwen3.8-27B-Q4_0.gguf}"
|
||||
: "${MMPROJ_FILE:=mmproj-Qwen3.8-27B-BF16.gguf}"
|
||||
: "${ALIAS:=frank-default-large}"
|
||||
|
||||
@@ -11,14 +11,14 @@ UNIT_FILE="$REPO_ROOT/scripts/frank-default-large.service"
|
||||
TEMPLATE_FILE="$REPO_ROOT/scripts/chat-template.jinja"
|
||||
|
||||
@test "frank-default-large serves the kingjones AGENT model" {
|
||||
grep -q '^MODEL_FILE=Qwen3\.8-27B-Q8_0_ROCMFPX_AGENT\.gguf$' "$ENV_FILE"
|
||||
grep -q '^MODEL_FILE=Qwen3\.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT\.gguf$' "$ENV_FILE"
|
||||
grep -q '^MTP_DRAFT_FILE=' "$ENV_FILE"
|
||||
! grep -q -- '--kairic-edge' "$SERVE_FILE"
|
||||
}
|
||||
|
||||
@test "frank-default-large uses the ROCmFPX runtime" {
|
||||
grep -q 'ROCmFPX/.*/bin/llama-server' "$ENV_FILE"
|
||||
grep -q 'build-kairic/bin/llama-server' "$ENV_FILE"
|
||||
grep -q 'build-promptforge/bin/llama-server' "$ENV_FILE"
|
||||
}
|
||||
|
||||
@test "frank-default-large runs standalone MTP" {
|
||||
|
||||
Reference in New Issue
Block a user