frank-large: document the ik_llama.cpp deployment and the decode performance campaign
- Runbook with the measured envelope (prefill ~192 t/s at the VNNI ceiling; decode ~3.8-4.3 t/s gated by 100 backend splits/token, GPU 95% idle), the root-cause evidence chain, tuning levers, and co-tenancy traps - AGENTS.md: fleet topology row + Active Services section for the ik deployment; halogen demoted to rollback; bench policy unit type fixed - LLM + root indexes relinked (frank-large-ik-uncensored ACTIVE, halogen-flash-server retired)
This commit is contained in:
1 parent
0c1e10a5ef
commit
795b0b3a5e
4 files changed
+164
-43
No files matched your search
@@ -1,10 +1,10 @@
|
||||
# LLM Fleet — Knowledge Base
|
||||
|
||||
Documentation for the LLM inference fleet on the two hosts
|
||||
(2026-09-15): charlie12345/ROCmFPX llama.cpp on Dankserver (frank-medium,
|
||||
(2026-09-16): charlie12345/ROCmFPX llama.cpp on Dankserver (frank-medium,
|
||||
kingjones777 Qwen3.8-27B-ROCmFPX-Q8_0-AGENT on 2x Radeon AI PRO R9700) and
|
||||
halogen-flash-server 0.5.8 on Dankclaw (frank-large, peonist-ai
|
||||
Qwen3.8-Flash-Next w4b).
|
||||
ik_llama.cpp on Dankclaw (frank-large, mradermacher Qwen3.8-Flash-Next-
|
||||
Uncensored i1-Q4_K_M on the RTX 3090 + Strix Halo hybrid).
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||||
@@ -26,11 +26,12 @@ layer split, 131k ctx / 1 slot, MTP]
|
||||
end
|
||||
|
||||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||||
E2[gfx1151 iGPU 64 GiB unified memory]
|
||||
E[halogen-flash-server 0.5.8
|
||||
E2[RTX 3090 (PCIe x4 dock) +
|
||||
gfx1151 iGPU 64 GiB unified memory]
|
||||
E[ik_llama.cpp master d85a1d7
|
||||
frank-large, port 7770]
|
||||
F[peonist-ai Qwen3.8-Flash-Next
|
||||
w4b .hgn + overlay, 262k ctx / 4 slots]
|
||||
F[mradermacher Qwen3.8-Flash-Next-Uncensored
|
||||
i1-Q4_K_M GGUF, 262k ctx / 1 slot, vision]
|
||||
E2 --> E --> F
|
||||
end
|
||||
|
||||
@@ -58,8 +59,13 @@ w4b .hgn + overlay, 262k ctx / 4 slots]
|
||||
(bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
|
||||
- [[deployment]] — deploy and update process for both servers
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [[halogen-flash-server]] — ACTIVE frank-large runbook since 2026-09-11:
|
||||
peonist-ai/halogen-flash-server 0.5.8 serving
|
||||
- [[frank-large-ik-uncensored]] — ACTIVE frank-large runbook since
|
||||
2026-09-15: ik_llama.cpp (root systemd unit) serving mradermacher/
|
||||
Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090 (dense/attention/
|
||||
KV) + LPDDR (experts/PLE) hybrid; measured performance envelope, the
|
||||
decode root-cause chain, tuning levers, and the co-tenancy traps
|
||||
- [[halogen-flash-server]] — RETIRED 2026-09-15 (rollback target): the
|
||||
peonist-ai halogen-flash-server 0.5.8 runbook serving
|
||||
peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn +
|
||||
quality overlay), rootless Quadlet container on Dankclaw port 7770 —
|
||||
deployed flags, memory model, client contract, troubleshooting
|
||||
|
||||
Reference in new issue
Block a user