dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs: update AGENTS.md LLM fleet topology for Tiel-Coder + Flash-Next swap

Browse Source
This commit is contained in:
Frank
2026-08-26 20:49:16 -07:00
parent 215d74f6ce
commit 2cd30de542
1 changed files with 37 additions and 14 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AGENTS.md
+37 -14
View File
@@ -36,8 +36,11 @@ then was reverted the same day for speed. On 2026-08-22 the service briefly
swapped to the Kairic Edge IU4 runtime, then was reverted to v3 the same day
(operator decision). On 2026-08-25 the service switched to the kingjones
Qwen3.8-27B-Uncensored-Q6_0-ROCmFPX_AGENT model (standalone MTP draft,
262144 context, reasoning on); the previous Q8_0 AGENT model was retired after
its `set_rows` crash. Runbooks:
262144 context, reasoning on). On 2026-09-01 both services were swapped:
Dankserver switched from ik_llama.cpp Ornith CRACK Q8_0 to stock llama.cpp
Tiel-Coder-35B (qwen35moe hybrid MoE), and Dankclaw switched from the Q6_0
AGENT model to the Flash-Next 56B MoE (qwen4exp, PR #27742 merged into
charlie's ROCmFPX fork, --fit off, 128k context). Runbooks:
[docs/LLM/index.md](docs/LLM/index.md).
Retired scripts live in `old/` (reference only) and `.AGENTTRASH/`.
@@ -262,23 +265,43 @@ When you rename a service or move a model to a new host, update
## Runtime & Troubleshooting
### kingjones Qwen3.8 Uncensored Q6_0 ROCmFPX AGENT llama.cpp (Dankclaw, ACTIVE)
### Qwen3.8-Flash-Next ROCmFP4 STRIX_LEAN llama.cpp (Dankclaw, ACTIVE)
The large service serves `kingjones/Qwen3.8-Uncensored-ROCmFP4-STRIX-MTP`
(public) from `/home/dank/models/kingjones-Qwen3.8-27B-STRIX-MTP/` with a
charlie12345/ROCmFPX `llama-server` (build `build-promptforge`). It runs a
standalone MTP draft head on ROCm0, f16 K + TurboQuant V KV cache, a full
262144 context / 1 slot, and reasoning on (Sharp template, budget -1,
`--reasoning-format none`; chat kwargs keep `preserve_thinking` +
`preserve_reasoning`). The Q6_0 main model is 24.86 GB. Env
`scripts/frank-default-large.env`, serve `scripts/serve-frank-large.sh`, and
runbook `docs/LLM/kingjones-q8agent-rocmfpx.md` define the deployment.
The large service serves
`kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX_LEAN-GGUF` (public, 98.5 GiB)
from `/home/dank/models/kingjones-Qwen3.8-Flash-Next-ROCmFP4-STRIX_LEAN/`
with a charlie12345/ROCmFPX `llama-server` built at
`/home/dank/GitPackages/ROCmFPX/build-qwen38` (PR #27742 qwen4exp architecture
merged). The model is a 56B MoE (512 experts, 10 used), 48 layers (12 attn +
36 SSM, full_attention_interval=4), QSA indexer (4 heads, top_k=2048),
hyper-connections (count=4, low_rank=320), PLE (layer 1). Q4_0_ROCMFP4 quant.
It runs f16 KV cache, 131072 context / 1 slot, `--fit off` (auto-fit segfaults
on qwen4exp hyper-connection config), and reasoning on (Sharp template, budget
-1, `--reasoning-format deepseek`; chat kwargs keep `preserve_thinking` +
`preserve_reasoning`). Env `scripts/frank-default-large.env`, serve
`scripts/serve-frank-large.sh`.
**Serving status (2026-09-01):** restored the previously deployed Q6_0 AGENT
model. The Q8_0 AGENT experiment is documented in
**Serving status (2026-09-01):** deployed the Flash-Next 56B MoE model. The
previous kingjones Q6_0 AGENT model (27B, standalone MTP, 262144 context) is
retired. The Q8_0 AGENT experiment is documented in
`docs/LLM/kingjones-q8agent-rocmfpx.md`; it was retired because it crashed in
`ggml_compute_forward_set_rows` and did not provide a working service.
### Tiel-Coder-35B-A3B llama.cpp (Dankserver, ACTIVE)
The small service serves `peculiar-ragdoll/Tiel-Coder-35B-A3B-GGUF` (public,
30 GB, UD-Q6_K_XL quant) from `/home/dank/models/tiel-coder-35b/` with
upstream llama.cpp (latest master, CUDA build with
`-DGGML_CUDA_FA_ALL_QUANTS=ON -DCMAKE_CUDA_ARCHITECTURES=86`). The model is a
qwen35moe hybrid MoE: 40 layers (10 attention + 30 SSM/Mamba,
full_attention_interval=4), 256 experts / 8 used, 2 KV heads × 256 dim. It
runs Q8_0 KV cache, 200700 total context (66900/slot × 3 parallel), `--cpu-moe`
for expert offload, `-ngl 999` (non-expert on GPU), and mmap. Env
`scripts/frank-default-small.env`, serve `scripts/serve-frank-small.sh`.
**Serving status (2026-09-01):** deployed the Tiel-Coder-35B model. The
previous ik_llama.cpp Ornith CRACK Q8_0 deployment is retired.
### charlie12345/ROCmFPX ROCmI4 W4A4 llama.cpp (Dankclaw, prior v3, 2026-08-18 to 2026-08-25)
The v4 runtime serves `cafonz/Qwen3.8-27B-ROCmI4-MTP-GGUF` (`eemin` namespace,