Restore Q6 AGENT large model
This commit is contained in:
1 parent
205b11aae4
commit
86d5f38142
8 files changed
+51
-57
No files matched your search
@@ -27,7 +27,7 @@ host.
|
||||
| Target | Runtime | Service | Model | Port |
|
||||
|--------|---------|---------|-------|------|
|
||||
| Dankserver | llama.cpp (upstream b10121) | frank-default-small | HauhauCS Qwen3.6-35B-A3B-Uncensored-Aggressive Q6_K_P + ngram-mod, 36 MoE experts on CPU | 7770 |
|
||||
| Dankclaw | charlie12345/ROCmFPX llama.cpp | frank-default-large | kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT Q8_0_ROCMFPX (standalone MTP draft head, Turbo4 V cache, 262K context / 1 slot), reasoning on | 7770 |
|
||||
| Dankclaw | charlie12345/ROCmFPX llama.cpp | frank-default-large | kingjones Qwen3.8 Uncensored Q6_0_ROCmFPX_AGENT (standalone MTP draft head, Turbo4 V cache, 262K context / 1 slot), reasoning on | 7770 |
|
||||
|
||||
History: the 2026-08-17 redo retired the pipeline-parallel split and the
|
||||
FastFlowLM NPU endpoint. The 2026-08-18 swap retired the vLLM service on
|
||||
@@ -35,8 +35,9 @@ Dankclaw. On 2026-08-21 a DFlash2 Vulkan deployment briefly replaced CIRU,
|
||||
then was reverted the same day for speed. On 2026-08-22 the service briefly
|
||||
swapped to the Kairic Edge IU4 runtime, then was reverted to v3 the same day
|
||||
(operator decision). On 2026-08-25 the service switched to the kingjones
|
||||
Qwen3.8-27B-ROCmFPX-Q8_0-AGENT model (standalone MTP draft, 262144 context,
|
||||
reasoning on); the previous Kairic Edge model was retired the same day. Runbooks:
|
||||
Qwen3.8-27B-Uncensored-Q6_0-ROCmFPX_AGENT model (standalone MTP draft,
|
||||
262144 context, reasoning on); the previous Q8_0 AGENT model was retired after
|
||||
its `set_rows` crash. Runbooks:
|
||||
[docs/LLM/index.md](docs/LLM/index.md).
|
||||
Retired scripts live in `old/` (reference only) and `.AGENTTRASH/`.
|
||||
|
||||
@@ -182,29 +183,22 @@ When you rename a service or move a model to a new host, update
|
||||
|
||||
## Runtime & Troubleshooting
|
||||
|
||||
### kingjones Qwen3.8-27B-ROCmFPX-Q8_0-AGENT llama.cpp (Dankclaw, ACTIVE)
|
||||
### kingjones Qwen3.8 Uncensored Q6_0 ROCmFPX AGENT llama.cpp (Dankclaw, ACTIVE)
|
||||
|
||||
The large service serves `kingjones/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT`
|
||||
(public) from `/home/dank/models/kingjones-Qwen3.8-27B-ROCMFPX-Q8_0-AGENT/`
|
||||
with a charlie12345/ROCmFPX `llama-server` (build `build-kairic`). It runs a
|
||||
The large service serves `kingjones/Qwen3.8-Uncensored-ROCmFP4-STRIX-MTP`
|
||||
(public) from `/home/dank/models/kingjones-Qwen3.8-27B-STRIX-MTP/` with a
|
||||
charlie12345/ROCmFPX `llama-server` (build `build-promptforge`). It runs a
|
||||
standalone MTP draft head on ROCm0, f16 K + TurboQuant V KV cache, a full
|
||||
262144 context / 1 slot, and reasoning on (Sharp template, budget -1,
|
||||
`--reasoning-format none`; chat kwargs keep `preserve_thinking` +
|
||||
`preserve_reasoning`). The payload (~28 GB main + 1.7 GB draft + 0.9 GB
|
||||
mmproj, ~30 GB total) plus the 262k KV cache fit in the lifted 114 GiB TTM/GTT
|
||||
budget (raised 2026-08-25). Env `scripts/frank-default-large.env`, serve
|
||||
`scripts/serve-frank-large.sh`, runbook
|
||||
[docs/LLM/kingjones-q8agent-rocmfpx.md](docs/LLM/kingjones-q8agent-rocmfpx.md).
|
||||
`preserve_reasoning`). The Q6_0 main model is 24.86 GB. Env
|
||||
`scripts/frank-default-large.env`, serve `scripts/serve-frank-large.sh`, and
|
||||
runbook `docs/LLM/kingjones-q8agent-rocmfpx.md` define the deployment.
|
||||
|
||||
**Serving status (2026-08-25):** the model loads but crashes in
|
||||
`ggml_compute_forward_set_rows` (NULL dispatch in `libggml-cpu.so.0`) before it
|
||||
can serve. Tested on all five available ROCmFPX builds (build-kairic v81,
|
||||
build-promptforge v3, build-rocmi4-w4a4 v257, build-strix-rocmfp4 v209) and the
|
||||
all-CPU path; all crash. Not GPU-specific, not MTP-specific, not fixed by
|
||||
`LLAMA_SET_ROWS=0`. Fix is a rebuild of charlie12345/ROCmFPX with a working
|
||||
`set_rows`. The proven serving context is 131072 at `gl=45` (262144 OOMs on
|
||||
device-local at `gl=999`). Config goals met: 114 GiB TTM/GTT, reasoning on,
|
||||
`preserve_thinking`/`preserve_reasoning` on.
|
||||
**Serving status (2026-09-01):** restored the previously deployed Q6_0 AGENT
|
||||
model. The Q8_0 AGENT experiment is documented in
|
||||
`docs/LLM/kingjones-q8agent-rocmfpx.md`; it was retired because it crashed in
|
||||
`ggml_compute_forward_set_rows` and did not provide a working service.
|
||||
|
||||
### charlie12345/ROCmFPX ROCmI4 W4A4 llama.cpp (Dankclaw, prior v3, 2026-08-18 to 2026-08-25)
|
||||
|
||||
|
||||
Reference in new issue
Block a user