docs: frank-large Q6 uncensored cutover + frank-small 2x200704 f16-KV reslice
- new active runbook frank-large-rocmfpx-q6agent-uncensored-mtp.md: the Q6_0_ROCMFPX_AGENT file (ftype 114, 339 type-102 + 159 type-103 tensors), 27.6-28.1 t/s at 86.8% draft acceptance, the measured ngram-mod chain rejection (-15% prose: recurrent-state rollback disabled), the bench-window hardening laws - frank-large-rocmfpx-q8agent-mtp.md retired; rollback section corrected (gufo payloads removed from disk 2026-10-03 - that rollback is now a re-download; the aligned Q8 file is the direct one, still on disk) - frank-small-llama-mimo-q5km.md reshaped to 2 x 200704 with fully unquantized f16 KV: boot-pair table gains the measured row (20,530 used / 4,046 free), 104.84 tok/s, slot-window contract at 200704, rollback to the 3x180224 + q8_0-V shape documented - AGENTS.md: fleet table rows, both endpoint sections, the chain law extended with the recurrent-model measurement, the --parallel law with the reslice, runbook index (active + retired) - LLM/index.md + root index entries updated
This commit is contained in:
@@ -1,6 +1,6 @@
|
||||
# LLM Fleet — Knowledge Base
|
||||
|
||||
Documentation for the LLM inference fleet on the two hosts (2026-10-02):
|
||||
Documentation for the LLM inference fleet on the two hosts (2026-10-03):
|
||||
**Dankserver runs the tcclaviger RDNA4 vLLM fork on frank-primary** (the
|
||||
digest-pinned `tcclaviger/vllm` image driving `podman run` directly, serving
|
||||
orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both Radeon AI PRO R9700s
|
||||
@@ -15,11 +15,12 @@ mounted cache dir** (the image ships no `/root/.cache` at all; unmounted, every
|
||||
SEQS 2 cannot compile this model. Rollback frank-primary-fp8-mtp-20261001),
|
||||
and Dankclaw
|
||||
runs two endpoints — **ROCmFPX llama.cpp (frank-large, port 7770:
|
||||
kingjones777/Qwen3.8-27B-ROCmFP4-STRIX-MTP-GGUF Q8_0_ROCMFPX_AGENT, a ftype-115
|
||||
file only that fork can read, with the repo's MTP head drafting 4 tokens, ONE
|
||||
session at 262,144, f16 KV, vision via the bundled projector)** and
|
||||
kingjones777/Qwen3.8-27B-Uncensored-ROCmFP4-STRIX-MTP-GGUF
|
||||
Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT, a ftype-114 file only that fork can
|
||||
read, with the repo's MTP head drafting 4 tokens at 27.6-28.1 t/s decode, ONE
|
||||
session at 262,144, f16 KV, vision via the BF16 projector)** and
|
||||
**llama.cpp CUDA (frank-small, port 7771: MiMo-V2.6-Distill-Qwen-9B Q5_K_M
|
||||
GGUF, 3 x 180,224, asymmetric KV f16 K + q8_0 V, on the RTX 3090)**.
|
||||
GGUF, 2 x 200,704, fully unquantized f16 KV, on the RTX 3090)**.
|
||||
|
||||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||||
@@ -46,12 +47,12 @@ expert-precise-memory arena 7,522 slots, /root/.cache mounted]
|
||||
gfx1151 iGPU 114 GiB GTT]
|
||||
E[ROCmFPX llama.cpp gfx1151, pinned fork build
|
||||
frank-large, port 7770]
|
||||
F[Qwen3.8-27B Q8_0_ROCMFPX_AGENT + MTP-Q4_0 head
|
||||
F[Qwen3.8-27B Uncensored Q6_0_ROCMFPX_AGENT + MTP-Q4_0 head
|
||||
1 x 262144 ctx, f16 KV, draft n-max 4, mmproj vision]
|
||||
G[llama.cpp master CUDA sm_86
|
||||
frank-small, port 7771]
|
||||
H[MiMo-V2.6 Q5_K_M GGUF
|
||||
3 x 180224 ctx, f16 K + q8_0 V, flash-attn, mmproj vision]
|
||||
2 x 200704 ctx, f16 K + f16 V, flash-attn, mmproj vision]
|
||||
E2 --> E --> F
|
||||
E2 --> G --> H
|
||||
end
|
||||
@@ -70,20 +71,23 @@ frank-small, port 7771]
|
||||
|
||||
## Active Deployment
|
||||
|
||||
- [[frank-large-rocmfpx-q8agent-mtp]] — **ACTIVE since 2026-10-02**
|
||||
- [[frank-large-rocmfpx-q6agent-uncensored-mtp]] — **ACTIVE since 2026-10-03**
|
||||
(root systemd unit, port 7770): the pinned kingjones30/ROCmFPX fork serving
|
||||
the repo's highest-quality rung `Qwen3.8-27B-Q8_0_ROCMFPX_AGENT.gguf`
|
||||
(26.28 GiB, ftype 115 = 187 plain-Q8_0 + 311 GGML_TYPE_Q8_0_ROCMFPX
|
||||
tensors) with the bundled `mtp-Qwen3.8-27B-Q4_0.gguf` head drafting 4
|
||||
tokens, one 262,144 window, f16 KV, the BF16 projector for vision. Why the
|
||||
runtime is not interchangeable, the pinned-build set_rows history, the
|
||||
measured envelope, rollback to gufo
|
||||
- [[frank-small-llama-mimo-q5km]] — **ACTIVE since 2026-10-02** (root
|
||||
systemd unit, port 7771): llama.cpp master (CUDA sm_86) serving
|
||||
bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF Q5_K_M on the RTX 3090,
|
||||
THREE slots of 180,224 tokens (pool 540,672) with K f16 + V q8_0 and the
|
||||
mmproj for vision, ~100 tok/s decode, ~2.7 GiB of the card left free.
|
||||
The KV arithmetic, the flash-attn dependency, rollback to Q6_K
|
||||
the uncensored protected-heads rung `Qwen3.8-27B-Uncensored-Q6_0_ROCMFPX_AGENT.gguf`
|
||||
(23.15 GiB, ftype 114 = 159 type-103 + 339 GGML_TYPE_Q6_0_ROCMFPX tensors,
|
||||
64 protected output heads) with the repo's `mtp-Qwen3.8-27B-Q4_0.gguf` head
|
||||
drafting 4 tokens, one 262,144 window, f16 KV, the BF16 projector for
|
||||
vision. Measured 27.6-28.1 t/s prose decode at 86.8 % draft acceptance; the
|
||||
ngram-mod chain onto MTP was measured and REJECTED (-15 %: it disables the
|
||||
hybrid's recurrent state rollback). Rollback: the aligned Q8 file, still on
|
||||
disk
|
||||
- [[frank-small-llama-mimo-q5km]] — **ACTIVE since 2026-10-02, resliced
|
||||
2026-10-03** (root systemd unit, port 7771): llama.cpp master (CUDA sm_86)
|
||||
serving bartowski/MiMo-V2.6-Distill-Qwen-9B-GGUF Q5_K_M on the RTX 3090,
|
||||
TWO slots of 200,704 tokens (pool 401,408) with FULLY UNQUANTIZED KV (f16 K
|
||||
+ f16 V) and the mmproj for vision, 104.8 tok/s decode, 4,046 MiB of the
|
||||
card left free. The boot-pair pool arithmetic, the slot-window contract,
|
||||
rollback to 3 x 180,224 or Q6_K
|
||||
- [[dankserver-qwen38-fp8-expert-offload-tuning]] — **ACTIVE since
|
||||
2026-10-02 (shape E)** (root systemd unit, port 7770): the tcclaviger RDNA4
|
||||
vLLM fork serving orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 across both
|
||||
@@ -192,8 +196,13 @@ frank-small, port 7771]
|
||||
(bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
|
||||
- [[deployment]] — deploy and update process for both servers
|
||||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||||
- [[frank-large-gufo-qwen38-27b]] — **RETIRED 2026-10-02 (the rollback
|
||||
target)** (root systemd unit, port 7770): gufo
|
||||
- [[frank-large-rocmfpx-q8agent-mtp]] — **RETIRED 2026-10-02 → 10-03 (the
|
||||
rollback target; payload still on disk)**: the same pinned ROCmFPX fork
|
||||
serving the ALIGNED `Q8_0_ROCMFPX_AGENT` (ftype 115, 17.0 t/s, MTP
|
||||
acceptance 0.462) — the MTP-vs-none +123 % A/B, the trap catalogue
|
||||
- [[frank-large-gufo-qwen38-27b]] — **RETIRED 2026-10-02 (history; the
|
||||
payload dirs were removed from disk 2026-10-03, so this rollback is a
|
||||
re-download)** (root systemd unit, port 7770): gufo
|
||||
(gufo-org/gufo) v0.1.1, the Strix Halo inference engine, built from
|
||||
source on Dankclaw, serving unsloth/Qwen3.8-27B-GGUF **UD-Q4_K_XL**
|
||||
with the z-lab DFlash2 Q4_K_M drafter — ONE session at the full native
|
||||
|
||||
Reference in New Issue
Block a user