Restore Q6 AGENT large model

This commit is contained in:
Frank committed 2026-08-25 13:36:07 -07:00
1 parent 205b11aae4
commit 86d5f38142
8 files changed
+51 -57

No files matched your search

+15 -21
View File
@@ -27,7 +27,7 @@ host.
| Target | Runtime | Service | Model | Port |
|--------|---------|---------|-------|------|
| Dankserver | llama.cpp (upstream b10121) | frank-default-small | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU | 7770 |
| Dankclaw | charlie12345/ROCmFPX llama.cpp | frank-default-large | kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT Q8_0_ROCMFPX (standalone MTP draft head, Turbo4 V cache, 262K context / 1 slot), reasoning on | 7770 |
| Dankclaw | charlie12345/ROCmFPX llama.cpp | frank-default-large | kingjones Qwen3.8 Uncensored Q6_0_ROCmFPX_AGENT (standalone MTP draft head, Turbo4 V cache, 262K context / 1 slot), reasoning on | 7770 |
History: the 2026-08-17 redo retired the pipeline-parallel split and the
FastFlowLM NPU endpoint. The 2026-08-18 swap retired the vLLM service on
@@ -35,8 +35,9 @@ Dankclaw. On 2026-08-21 a DFlash2 Vulkan deployment briefly replaced CIRU,
then was reverted the same day for speed. On 2026-08-22 the service briefly
swapped to the Kairic Edge IU4 runtime, then was reverted to v3 the same day
(operator decision). On 2026-08-25 the service switched to the kingjones
Qwen3.8-27B-ROCmFPX-Q8_0-AGENT model (standalone MTP draft, 262144 context,
reasoning on); the previous Kairic Edge model was retired the same day. Runbooks:
Qwen3.8-27B-Uncensored-Q6_0-ROCmFPX_AGENT model (standalone MTP draft,
262144 context, reasoning on); the previous Q8_0 AGENT model was retired after
its `set_rows` crash. Runbooks:
[docs/LLM/index.md](docs/LLM/index.md).
Retired scripts live in `old/` (reference only) and `.AGENTTRASH/`.
@@ -182,29 +183,22 @@ When you rename a service or move a model to a new host, update
## Runtime & Troubleshooting
### kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT llama.cpp (Dankclaw, ACTIVE)
### kingjones Qwen3.8 Uncensored Q6_0 ROCmFPX AGENT llama.cpp (Dankclaw, ACTIVE)
The large service serves `kingjones/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT`
(public) from `/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT/`
with a charlie12345/ROCmFPX `llama-server` (build `build-kairic`). It runs a
The large service serves `kingjones/Qwen3.8-Uncensored-ROCmFP4-STRIX-MTP`
(public) from `/home/dank/models/kingjones-Qwen3.8-27B-STRIX-MTP/` with a
charlie12345/ROCmFPX `llama-server` (build `build-promptforge`). It runs a
standalone MTP draft head on ROCm0, f16 K + TurboQuant V KV cache, a full
262144 context / 1 slot, and reasoning on (Sharp template, budget -1,
`--reasoning-format none`; chat kwargs keep `preserve_thinking` +
`preserve_reasoning`). The payload (~28 GB main + 1.7 GB draft + 0.9 GB
mmproj, ~30 GB total) plus the 262k KV cache fit in the lifted 114 GiB TTM/GTT
budget (raised 2026-08-25). Env `scripts/frank-default-large.env`, serve
`scripts/serve-frank-large.sh`, runbook
[docs/LLM/kingjones-q8agent-rocmfpx.md](docs/LLM/kingjones-q8agent-rocmfpx.md).
`preserve_reasoning`). The Q6_0 main model is 24.86 GB. Env
`scripts/frank-default-large.env`, serve `scripts/serve-frank-large.sh`, and
runbook `docs/LLM/kingjones-q8agent-rocmfpx.md` define the deployment.
**Serving status (2026-08-25):** the model loads but crashes in
`ggml_compute_forward_set_rows` (NULL dispatch in `libggml-cpu.so.0`) before it
can serve. Tested on all five available ROCmFPX builds (build-kairic v81,
build-promptforge v3, build-rocmi4-w4a4 v257, build-strix-rocmfp4 v209) and the
all-CPU path; all crash. Not GPU-specific, not MTP-specific, not fixed by
`LLAMA_SET_ROWS=0`. Fix is a rebuild of charlie12345/ROCmFPX with a working
`set_rows`. The proven serving context is 131072 at `gl=45` (262144 OOMs on
device-local at `gl=999`). Config goals met: 114 GiB TTM/GTT, reasoning on,
`preserve_thinking`/`preserve_reasoning` on.
**Serving status (2026-09-01):** restored the previously deployed Q6_0 AGENT
model. The Q8_0 AGENT experiment is documented in
`docs/LLM/kingjones-q8agent-rocmfpx.md`; it was retired because it crashed in
`ggml_compute_forward_set_rows` and did not provide a working service.
### charlie12345/ROCmFPX ROCmI4 W4A4 llama.cpp (Dankclaw, prior v3, 2026-08-18 to 2026-08-25)