Frank
87caca4894
AGENTS.md topology, Active Services, the CUDA-quant-KV lesson (f16 is no longer why), and the runbook index. New frank-small-gemma4-12b runbook: payload digests, the SWA/q8 arithmetic behind --fit's 262k window, the MTP-sidecar + ngram-mod chain, the always-accepted reasoning_effort, and the fit-probe error that looks like a fault at boot. The Qwen runbook and both indexes carry the retirement; work log updated.
201 lines
10 KiB
Markdown
201 lines
10 KiB
Markdown
# LLM Fleet — Knowledge Base
|
||
|
||
Documentation for the LLM inference fleet on the two hosts (2026-09-16):
|
||
**mainline llama.cpp on Dankserver** serving frank-medium as one layer-split
|
||
process across both Radeon AI PRO R9700 cards (ajgazin Swift-Qwen3.8-27B-
|
||
Uncensored-Dynamic-MTP UD-Q8_K_XL + the z-lab DFlash2 drafter), and TWO
|
||
endpoints on Dankclaw — halogen-flash-server (frank-large, Qwen3.8-Flash-Next
|
||
w4b on the gfx1151 iGPU, port 7770) and mainline llama.cpp (frank-small,
|
||
ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S with MTP and vision on the RTX 3090, port
|
||
7771).
|
||
|
||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||
> `serverAppConfigs-docs/` root to `serverAppConfigs-docs/LLM/`. The fleet scripts and units now live in the
|
||
> repo's `scripts/` directory.
|
||
|
||
## Deployment Topology
|
||
|
||
```mermaid
|
||
graph LR
|
||
subgraph Dankserver[AMD Dankserver - Ubuntu 26.04]
|
||
A[2x Radeon AI PRO R9700
|
||
gfx1201, 32 GiB each]
|
||
B2[llama.cpp, layer split
|
||
frank-medium, port 7770]
|
||
C[Swift-Qwen3.8-27B-Uncensored UD-Q8_K_XL
|
||
ONE process on both cards, 262144 ctx / 1 slot, DFlash2]
|
||
A --> B2 --> C
|
||
end
|
||
|
||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||
E2[RTX 3090 (PCIe x4 dock) +
|
||
gfx1151 iGPU 64 GiB unified memory]
|
||
E[halogen-flash-server 0.5.8
|
||
frank-large, port 7770]
|
||
F[peonist-ai Qwen3.8-Flash-Next w4b
|
||
2 x 200192 ctx over one 400384-position pool]
|
||
G[mainline llama.cpp 37b53fd
|
||
frank-small, port 7771]
|
||
H[Qwen3.8-27B GSQ-RCO IQ3_S
|
||
131072 ctx / 1 slot, MTP + vision]
|
||
E2 --> E --> F
|
||
E2 --> G --> H
|
||
end
|
||
|
||
CLI[LLM Clients / API] -->|HTTP :7770| B2
|
||
CLI -->|HTTP :7770| E
|
||
CLI -->|HTTP :7771| G
|
||
|
||
## Servers
|
||
|
||
| Server | GPU | VRAM / Memory | OS |
|
||
| --------------------------------------- | ----------------------------- | -------------------- | ------------ |
|
||
| [[Dankserver System Specs\|Dankserver]] | 2x AMD Radeon AI PRO R9700 | 2x 32 GB VRAM, 60 GB RAM | Ubuntu 26.04 |
|
||
| [[Dankclaw System Specs\|Dankclaw]] | AMD Radeon 8060S (Strix Halo) | gfx1151 unified memory | Ubuntu 26.04 |
|
||
|
||
## Active Deployment
|
||
|
||
- [[frank-medium-llamacpp-dflash2]] — the ACTIVE frank-medium runbook since
|
||
2026-09-16: mainline llama.cpp serving
|
||
ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF at UD-Q8_K_XL with the
|
||
z-lab DFlash2 Q8_0 drafter on Dankserver's 2x Radeon AI PRO R9700, as ONE
|
||
layer-split process behind :7770 (tensor parallelism is impossible on this
|
||
host — the second card cannot dispatch an RCCL collective)
|
||
- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-16, one day after it was
|
||
deployed: the vllm-radiance MXFP4 stack (stilldeadcode/vllm-radiance:0.9.3 +
|
||
ggz14/radiance-vllm-mxfp4 patches) serving
|
||
just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq as two single-GPU replicas
|
||
behind a round-robin proxy; kept as the record of the deployment it
|
||
succeeded, plus the PCIe-atomics evidence that rules out TP=2 on this host
|
||
- [[frank-medium-rocmfpx]] — PRIOR frank-medium deployment (llama.cpp era,
|
||
2026-09-15): charlie12345/ROCmFPX serving
|
||
kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the
|
||
runtime it replaced
|
||
- [[freetoken-frank-small]] — RETIRED 2026-09-15, when the RTX 3090 moved to
|
||
frank-large: the FreeToken v0.1.2 frank-small runbook (batsclamp
|
||
Huihui-Qwen3.6-35B-A3B-abliterated-FP8), kept as the retirement record. The
|
||
NAME was reused on 2026-09-16 by a different service on Dankclaw — see
|
||
[[frank-small-qwen38-gsq-rco]]
|
||
- [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep
|
||
(bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
|
||
- [[deployment]] — deploy and update process for both servers
|
||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||
- [[halogen-flash-server]] — ACTIVE frank-large runbook again since 2026-09-16
|
||
(rootless Quadlet, port 7770): peonist-ai halogen-flash-server 0.5.8 serving
|
||
peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn + quality
|
||
overlay) on the gfx1151 iGPU — two 200k conversations over one
|
||
400,384-position pool, deployed flags, memory model, client contract,
|
||
troubleshooting
|
||
- [[frank-small-gemma4-12b]] — ACTIVE frank-small runbook since 2026-09-17
|
||
(root systemd unit, port 7771): mainline llama.cpp serving
|
||
HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX
|
||
3090, with the separate MTP assistant sidecar chained behind ngram-mod,
|
||
q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB
|
||
- [[frank-small-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target): the
|
||
prior frank-small runbook (2026-09-16 → 09-17), ISTA-DASLab
|
||
Qwen3.8-27B-GSQ-RCO IQ3_S with the model's own in-file MTP head and
|
||
f16 KV at 131k
|
||
- [[x4-pair-experiment]] — MEASURED 2026-09-16: what mainline llama.cpp can and
|
||
cannot do with the 3090 + Strix Halo pair, over their only link (the 3090's
|
||
Gen4 x4). The tensor inventory that decides any split, the single flag (`-ot`)
|
||
that kills per-tensor placement across devices, the two-device layer splits
|
||
that work (+7 % decode at 64k, cross-vendor included), the harness-parsing trap
|
||
that invalidated the first round, and the iGPU-only result that matches
|
||
halogen's deployed decode rate — at a memory cost the co-tenants cannot pay
|
||
- [[frank-large-ik-uncensored]] — RETIRED 2026-09-16 (rollback target): the
|
||
ik_llama.cpp runbook (root systemd unit, 2026-09-15 → 09-16) serving
|
||
mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090
|
||
(dense/attention/KV) + LPDDR (experts/PLE) hybrid; its measured envelope and
|
||
decode root-cause chain are the record of why the fleet went back to halogen
|
||
- [[dankclaw-iommu-translation-off]] — why Dankclaw boots with
|
||
`amd_iommu=off` since 2026-09-11 (13-16 % prefill, measured upstream),
|
||
what it cost (the NPU endpoint), what was deliberately not copied from
|
||
the reference machine, and how to revert
|
||
- [[kingjones-strix-rocmfp4]] — PRIOR frank-large runbook and rollback
|
||
target: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF, 177B params
|
||
(56B MoE + 51.2B PLE), single 113.47 GiB GGUF, on the card's pinned
|
||
kingjones30/ROCmFPX fork (2026-08-30 → 2026-09-11)
|
||
- [[ubatch-benchmark-large]] — active-model llama-bench ubatch sweep plan and results
|
||
- [[frank-medium-batch-ubatch]] — batch/ubatch sweep on the 2x R9700 layer split
|
||
(2026-09-16): the ubatch optimum at 512, why the logical batch is free, and
|
||
the split rebalance that followed it
|
||
- [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure
|
||
- [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B embedded-MTP model
|
||
on Dankclaw (retired)
|
||
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on
|
||
Dankserver (frank-small, retired 2026-09-02)
|
||
- [[ikllama-ornith-dankserver]] — the retired ik_llama.cpp runtime for
|
||
frank-small (2026-08-26): q8_0 + Hadamard KV cache,
|
||
split-mode none (single-GPU), mandatory FA_ALL_QUANTS build flags
|
||
- [[kingjones-strix-mtp]] — the former Q6 AGENT runbook (restored 2026-09-01
|
||
after Q8_0 failure, then superseded by the STRIX deployment on 2026-08-27)
|
||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime
|
||
on Dankclaw (superseded 2026-08-24 by the kingjones ROCmFP4 model)
|
||
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment
|
||
(deployed and reverted 2026-08-22)
|
||
- [[Dankserver System Specs]] — the 2x R9700 box (frank-medium)
|
||
- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large on
|
||
halogen-flash-server since 2026-09-16, frank-small on the RTX 3090)
|
||
- [[frank-npu]] — RETIRED 2026-09-11: the FLM NPU sight+sound endpoint
|
||
on Dankclaw (Gemma4 E4B vision+audio pack, XDNA2, port 7771), kept as
|
||
the retirement record — the IOMMU/SVA err=-19 root cause and where
|
||
every dependent route moved
|
||
|
||
## CIRU Reference (Dankclaw)
|
||
|
||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 model, ROCm 7.15 toolchain,
|
||
and build history (on Dankclaw; the slot now serves the kingjones
|
||
ROCmFP4 model, see [[kingjones-strix-mtp]])
|
||
- [[kairic-edge]] — Kairic Edge IU4 runtime: build runbook, binary
|
||
identity, and experiment summary (retired 2026-08-22)
|
||
|
||
## DFlash2 Reference (archived)
|
||
|
||
- [[dflash2-large-vulkan]] — the retired DFlash2 Vulkan deployment
|
||
(2026-08-21)
|
||
|
||
## vLLM Reference (historical)
|
||
|
||
- [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism
|
||
- [[VLLM Index]] — quick reference with commands, endpoints, and parameter table
|
||
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware
|
||
notes and sizing (this architecture family is what the retired frank-small
|
||
served in the Huihui abliterated FP8 variant — see
|
||
[[freetoken-frank-small]]; the Ornith-1.5-35B-A3B CRACK checkpoint
|
||
that preceded it is archived at [[dealign-ornith-crack]])
|
||
|
||
## Memory Management
|
||
|
||
- [[RAM CPU Offloading in VLLM]] — KV cache swap between GPU VRAM and system RAM
|
||
- [[Hybrid CPU GPU Processing Dankserver]] — NVIDIA GPU compute with system RAM offload
|
||
- [[Shared Memory Architecture Dankclaw]] — AMD unified memory and ROCm configuration
|
||
|
||
## Retired Topology (2026-08-17 redo)
|
||
|
||
- [[Qwen3.8 PP Deployment]] — the retired pipeline-parallel head + worker
|
||
- [[qwen3.8-spec-matrix]] — Qwen3.8-27B spec history (superseded by FastMTP)
|
||
- [[FastFlowLM NPU Inference]] — the retired FastFlowLM NPU endpoint
|
||
- [[frank-large-benchmark]] — backend/spec benchmark behind the old head
|
||
- [[dankserver-vram-growth]] — VRAM/RAM growth behind the retired cuBLAS crash
|
||
- [[DFlash DSpark Speculative Decoding]] — retired DFlash/DSpark drafting
|
||
- [[Quantization by Hardware Architecture]] — which quant format fits which GPU/CPU
|
||
|
||
## Pipeline Parallelism (archive)
|
||
|
||
- [[Pipeline Parallelism/index]] — index of multi-node inference research. The PP
|
||
pair is retired. These pages are research history.
|
||
|
||
## Runtime Research (llama.cpp family)
|
||
|
||
- [[Llama Cpp Runtimes]] — mainline vs charlie12345/ROCmFPX vs ik_llama.cpp: positioning, quantization, server fit
|
||
- [[Llama Cpp Code Diff]] — code-level quantization diff of the three runtimes + server recommendations
|
||
- [[SGLang]] — the structured-generation serving engine; when it fits Dankserver (2x R9700) vs Dankclaw (Strix Halo)
|
||
|
||
## Related
|
||
|
||
- [Project knowledge base](../index.md) — the Podman app stacks on the same
|
||
two hosts. OpenWebUI (Dankclaw) consumes the on-host `frank-large`
|
||
endpoint at `:7770`. The Gitea instance on Dankclaw hosts this repository.
|
||
- [Deployment Flow](../Architecture/Deployment%20Flow.md) — the service-scoped
|
||
git-based deploy with inference isolation |