docs: update AGENTS.md LLM fleet topology for Tiel-Coder + Flash-Next swap

This commit is contained in:
Frank committed 2026-08-26 20:49:16 -07:00
1 parent 215d74f6ce
commit 2cd30de542
1 file changed
+37 -14
+37 -14
View File
@@ -36,8 +36,11 @@ then was reverted the same day for speed. On 2026-08-22 the service briefly
swapped to the Kairic Edge IU4 runtime, then was reverted to v3 the same day swapped to the Kairic Edge IU4 runtime, then was reverted to v3 the same day
(operator decision). On 2026-08-25 the service switched to the kingjones (operator decision). On 2026-08-25 the service switched to the kingjones
Qwen3.8-27B-Uncensored-Q6_0-ROCmFPX_AGENT model (standalone MTP draft, Qwen3.8-27B-Uncensored-Q6_0-ROCmFPX_AGENT model (standalone MTP draft,
262144 context, reasoning on); the previous Q8_0 AGENT model was retired after 262144 context, reasoning on). On 2026-09-01 both services were swapped:
its `set_rows` crash. Runbooks: Dankserver switched from ik_llama.cpp Ornith CRACK Q8_0 to stock llama.cpp
Tiel-Coder-35B (qwen35moe hybrid MoE), and Dankclaw switched from the Q6_0
AGENT model to the Flash-Next 56B MoE (qwen4exp, PR #27742 merged into
charlie's ROCmFPX fork, --fit off, 128k context). Runbooks:
[docs/LLM/index.md](docs/LLM/index.md). [docs/LLM/index.md](docs/LLM/index.md).
Retired scripts live in `old/` (reference only) and `.AGENTTRASH/`. Retired scripts live in `old/` (reference only) and `.AGENTTRASH/`.
@@ -262,23 +265,43 @@ When you rename a service or move a model to a new host, update
## Runtime & Troubleshooting ## Runtime & Troubleshooting
### kingjones Qwen3.8 Uncensored Q6_0 ROCmFPX AGENT llama.cpp (Dankclaw, ACTIVE) ### Qwen3.8-Flash-Next ROCmFP4 STRIX_LEAN llama.cpp (Dankclaw, ACTIVE)
The large service serves `kingjones/Qwen3.8-Uncensored-ROCmFP4-STRIX-MTP` The large service serves
(public) from `/home/dank/models/kingjones-Qwen3.8-27B-STRIX-MTP/` with a `kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX_LEAN-GGUF` (public, 98.5 GiB)
charlie12345/ROCmFPX `llama-server` (build `build-promptforge`). It runs a from `/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX_LEAN/`
standalone MTP draft head on ROCm0, f16 K + TurboQuant V KV cache, a full with a charlie12345/ROCmFPX `llama-server` built at
262144 context / 1 slot, and reasoning on (Sharp template, budget -1, `/home/dank/GitPackages/ROCmFPX/build-qwen38` (PR #27742 qwen4exp architecture
`--reasoning-format none`; chat kwargs keep `preserve_thinking` + merged). The model is a 56B MoE (512 experts, 10 used), 48 layers (12 attn +
`preserve_reasoning`). The Q6_0 main model is 24.86 GB. Env 36 SSM, full_attention_interval=4), QSA indexer (4 heads, top_k=2048),
`scripts/frank-default-large.env`, serve `scripts/serve-frank-large.sh`, and hyper-connections (count=4, low_rank=320), PLE (layer 1). Q4_0_ROCMFP4 quant.
runbook `docs/LLM/kingjones-q8agent-rocmfpx.md` define the deployment. It runs f16 KV cache, 131072 context / 1 slot, `--fit off` (auto-fit segfaults
on qwen4exp hyper-connection config), and reasoning on (Sharp template, budget
-1, `--reasoning-format deepseek`; chat kwargs keep `preserve_thinking` +
`preserve_reasoning`). Env `scripts/frank-default-large.env`, serve
`scripts/serve-frank-large.sh`.
**Serving status (2026-09-01):** restored the previously deployed Q6_0 AGENT **Serving status (2026-09-01):** deployed the Flash-Next 56B MoE model. The
model. The Q8_0 AGENT experiment is documented in previous kingjones Q6_0 AGENT model (27B, standalone MTP, 262144 context) is
retired. The Q8_0 AGENT experiment is documented in
`docs/LLM/kingjones-q8agent-rocmfpx.md`; it was retired because it crashed in `docs/LLM/kingjones-q8agent-rocmfpx.md`; it was retired because it crashed in
`ggml_compute_forward_set_rows` and did not provide a working service. `ggml_compute_forward_set_rows` and did not provide a working service.
### Tiel-Coder-35B-A3B llama.cpp (Dankserver, ACTIVE)
The small service serves `peculiar-ragdoll/Tiel-Coder-35B-A3B-GGUF` (public,
30 GB, UD-Q6_K_XL quant) from `/home/dank/models/tiel-coder-35b/` with
upstream llama.cpp (latest master, CUDA build with
`-DGGML_CUDA_FA_ALL_QUANTS=ON -DCMAKE_CUDA_ARCHITECTURES=86`). The model is a
qwen35moe hybrid MoE: 40 layers (10 attention + 30 SSM/Mamba,
full_attention_interval=4), 256 experts / 8 used, 2 KV heads × 256 dim. It
runs Q8_0 KV cache, 200700 total context (66900/slot × 3 parallel), `--cpu-moe`
for expert offload, `-ngl 999` (non-expert on GPU), and mmap. Env
`scripts/frank-default-small.env`, serve `scripts/serve-frank-small.sh`.
**Serving status (2026-09-01):** deployed the Tiel-Coder-35B model. The
previous ik_llama.cpp Ornith CRACK Q8_0 deployment is retired.
### charlie12345/ROCmFPX ROCmI4 W4A4 llama.cpp (Dankclaw, prior v3, 2026-08-18 to 2026-08-25) ### charlie12345/ROCmFPX ROCmI4 W4A4 llama.cpp (Dankclaw, prior v3, 2026-08-18 to 2026-08-25)
The v4 runtime serves `cafonz/Qwen3.8-27B-ROCmI4-MTP-GGUF` (`eemin` namespace, The v4 runtime serves `cafonz/Qwen3.8-27B-ROCmI4-MTP-GGUF` (`eemin` namespace,