# LLM Fleet — Knowledge Base

Documentation for the LLM inference fleet on the two hosts (2026-09-21):
**vllm-radiance on Dankserver** serving frank-medium as ONE tensor-parallel
process across both Radeon AI PRO R9700 cards (just1moremodel
Qwen3.8-27B-Uncensored-MXFP4 + the z-lab DFlash2 drafter, bf16 KV, TP=2 over the
R4D one-shot allreduce), and TWO endpoints on Dankclaw — stock llama.cpp
(unsloth b11030, ROCm gfx1151; frank-large, unsloth Qwen3.8-27B UD-Q8_K_XL +
the z-lab DFlash2 Q4_K_M drafter on the gfx1151 iGPU, port 7770) and mainline
llama.cpp (frank-small, HauhauCS Gemma4-12B QAT Q4_K_M with MTP and vision on
the RTX 3090, port 7771).

> 2026-08-22: the former `vllmScripts` repository is merged into the
> serverAppConfigs repository. This knowledge base moved from the old repo's
> `serverAppConfigs-docs/` root to `serverAppConfigs-docs/LLM/`. The fleet scripts and units now live in the
> repo's `scripts/` directory.

## Deployment Topology

```mermaid
graph LR
    subgraph Dankserver[AMD Dankserver - Ubuntu 26.04]
        A[2x Radeon AI PRO R9700
gfx1201, 32 GiB each]
        B2[vllm-radiance 0.9.3, TP=2
frank-medium, port 7770]
        C[Qwen3.8-27B MXFP4 + DFlash2 drafter
ONE process on both cards, 262144 ctx, up to 8 seqs]
        A --> B2 --> C
    end

    subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
        E2[RTX 3090 (PCIe x4 dock) +
gfx1151 iGPU 114 GiB GTT]
        E[stock llama.cpp (unsloth b11030, gfx1151)
frank-large, port 7770]
        F[Qwen3.8-27B UD-Q8_K_XL + DFlash2 Q4_K_M drafter
262144 ctx / 1 slot, DFlash2 depth 4, q8_0 KV]
        G[mainline llama.cpp 37b53fd
frank-small, port 7771]
        H[Gemma4-12B QAT Q4_K_M
4 x 262144 ctx, MTP + vision + audio]
        E2 --> E --> F
        E2 --> G --> H
    end

    CLI[LLM Clients / API] -->|HTTP :7770| B2
    CLI -->|HTTP :7770| E
    CLI -->|HTTP :7771| G
```

## Servers

| Server                                  | GPU                           | VRAM / Memory        | OS           |
| --------------------------------------- | ----------------------------- | -------------------- | ------------ |
| [[Dankserver System Specs\|Dankserver]] | 2x AMD Radeon AI PRO R9700 | 2x 32 GB VRAM, 60 GB RAM | Ubuntu 26.04 |
| [[Dankclaw System Specs\|Dankclaw]]     | AMD Radeon 8060S (Strix Halo) | gfx1151 unified memory | Ubuntu 26.04 |

## Active Deployment

- [[frank-medium-radiance-mxfp4]] — the ACTIVE frank-medium runbook:
  vllm-radiance (stilldeadcode/vllm-radiance:0.9.3) serving
  just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq (MXFP4/quark W4A8) with the
  z-lab DFlash2-FP8 drafter as ONE tensor-parallel process (TP=2) behind
  :7770, now on unquantized bf16 KV (`KV_DTYPE=auto`, cut 2026-09-21 to test
  whether the endpoint's historical fp8 KV had been degrading W4A8 quality).
  The FP8 checkpoint (orcarouter/Qwen3.8-27B-Uncensored-FP8, the 09-20→09-21
  shape) is the on-disk rollback. The three-arm battery that chose TP=2, the
  auto-sized KV pool, and verified capabilities (tools, reasoning)
- [[frank-medium-model-swap]] — how to swap the checkpoint behind frank-medium:
  the streamlining toolchain (hf-manifest.py, hf-fetch.sh,
  swap-frank-medium.sh), the quantization-aware entrypoint contract, the
  quadlet edit, the deploy, the capability battery, and the rollback path.
  The 2026-09-20 MXFP4 → FP8 swap is the worked example
- [[frank-medium-llamacpp-dflash2]] — RETIRED 2026-09-16 → 2026-09-20: mainline
  llama.cpp serving orcarouter/Qwen3.8-27B-Uncensored-GGUF at Q8_0 with the
  DFlash2 Q4_K_M drafter as ONE layer-split process (32:33 over two
  163,840-token slots); superseded by the radiance TP=2 cutover, and the
  rollback target. Keeps the batch/ubatch optimum curve and the per-layer split
  arithmetic
- [[Dual-R9700-Cooperation]] — whether the two R9700s can serve ONE model: the
  RCCL/PCIe-atomics wall on the old TUF GAMING B650-E, the 2026-09-20 ProArt
  B850-Creator x8/x8 swap that cleared it, and the measured three-way verdict
  (radiance TP=2 95.8 t/s decode vs layer split 45.7 vs llama tensor mode 30.5)
  that put TP=2 in production
- [[frank-medium-rocmfpx]] — PRIOR frank-medium deployment (llama.cpp era,
  2026-09-15): charlie12345/ROCmFPX serving
  kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the
  runtime it replaced
- [[freetoken-frank-small]] — RETIRED 2026-09-15, when the RTX 3090 moved to
  frank-large: the FreeToken v0.1.2 frank-small runbook (batsclamp
  Huihui-Qwen3.6-35B-A3B-abliterated-FP8), kept as the retirement record. The
  NAME was reused on 2026-09-16 by a different service on Dankclaw — see
  [[frank-small-qwen38-gsq-rco]]
- [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep
  (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
- [[deployment]] — deploy and update process for both servers
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
- [[frank-large-unsloth-qwen38-27b-q8xl]] — ACTIVE frank-large runbook
  since 2026-09-21 (root systemd unit, port 7770): stock llama.cpp
  (unsloth b11030, ROCm gfx1151) serving
  unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL (27B dense) with the z-lab DFlash2
  Q4_K_M drafter (depth 4) on the gfx1151 iGPU — one full 262,144-token
  conversation, q8_0 KV (the build lacks fp8), text-only, Qwen-Sharp
  v22.4.0 template, the 114 GiB GTT memory model, client contract,
  troubleshooting
- [[frank-large-kairic-edge-qwen38-27b]] — RETIRED 2026-09-21 (same-day
  swap, replaced by [[frank-large-unsloth-qwen38-27b-q8xl]]; rollback
  target): the Kairic Edge runtime (ciru-ai/ROCmFPX v1.2, built from
  source) serving jcbtc/Qwen3.8-27B-IU4-Kairic-Edge (IU4 W4A4, 27B dense)
  on the gfx1151 iGPU — one full 262,144-token conversation, MTP depth 4 +
  n-gram drafting, the three PromptForge sidecars, compatibility mode, the
  114 GiB GTT memory model, client contract, troubleshooting
- [[frank-large-unsloth-qwen38-flash-next]] — RETIRED 2026-09-21 (same-day
  swap, replaced by [[frank-large-kairic-edge-qwen38-27b]]; rollback target):
  the unsloth llama.cpp prebuilt (b11030-mix-5ff778e, ROCm gfx1151) serving
  unsloth/Qwen3.8-Flash-Next-GGUF:UD-IQ4_XS (125B MoE) on the gfx1151 iGPU —
  one 200,000-token conversation, MTP depth 2 via the explicit -md shared-Q8_0
  head, the 114 GiB GTT memory model, client contract, troubleshooting
- [[frank-large-ciru-orca]] — RETIRED 2026-09-21 (replaced by
  [[frank-large-unsloth-qwen38-flash-next]]; rollback target): the CIRU
  v4.4.1 runtime (private host build, QSA hotfix) serving
  jcbtc/Qwen3.8-Flash-CIRU-STRIX-Orca w4b on the gfx1151 iGPU — one
  262,144-token conversation, MTP depth 4, 4096 MiB PLE cache, runtime-root
  gate, client contract, troubleshooting (2026-09-19 → 09-21)
- [[halogen-flash-server]] — RETIRED 2026-09-19 (replaced by
  [[frank-large-ciru-orca]]; rollback target): peonist-ai halogen-flash-server
  0.5.8 serving peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b
  .hgn + quality overlay) on the gfx1151 iGPU — two 200k conversations over one
  400,384-position pool, deployed flags, memory model, client contract,
  troubleshooting (2026-09-16 → 09-19)
- [[frank-small-gemma4-12b]] — ACTIVE frank-small runbook since 2026-09-17
  (root systemd unit, port 7771): mainline llama.cpp serving
  HauhauCS/Gemma4-12B-QAT-Uncensored-HauhauCS-Balanced Q4_K_M on the RTX
  3090, with the separate MTP assistant sidecar chained behind ngram-mod,
  q8_0 KV, and `--fit` sizing the full 262k window into ~12 GiB. A same-day
  vLLM detour (2026-09-19) was reverted the same day; see below.
- [[frank-small-vllm-gemma4]] — RETIRED 2026-09-19 (same-day revert,
  rollback target): vLLM serving google/gemma-4-12B-it-qat-w4a16-ct (w4a16
  Marlin) with the bf16 MTP assistant on the RTX 3090 — bf16 KV, 289,647-
  token pool, 2 concurrent full-context sessions, native text+image+audio.
- [[frank-small-qwen38-gsq-rco]] — RETIRED 2026-09-17 (rollback target): the
  prior frank-small runbook (2026-09-16 → 09-17), ISTA-DASLab
  Qwen3.8-27B-GSQ-RCO IQ3_S with the model's own in-file MTP head and
  f16 KV at 131k
- [[x4-pair-experiment]] — MEASURED 2026-09-16: what mainline llama.cpp can and
  cannot do with the 3090 + Strix Halo pair, over their only link (the 3090's
  Gen4 x4). The tensor inventory that decides any split, the single flag (`-ot`)
  that kills per-tensor placement across devices, the two-device layer splits
  that work (+7 % decode at 64k, cross-vendor included), the harness-parsing trap
  that invalidated the first round, and the iGPU-only result that matches
  halogen's deployed decode rate — at a memory cost the co-tenants cannot pay
- [[frank-large-ik-uncensored]] — RETIRED 2026-09-16 (rollback target): the
  ik_llama.cpp runbook (root systemd unit, 2026-09-15 → 09-16) serving
  mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090
  (dense/attention/KV) + LPDDR (experts/PLE) hybrid; its measured envelope and
  decode root-cause chain are the record of why the fleet went back to halogen
- [[dankclaw-iommu-translation-off]] — why Dankclaw boots with
  `amd_iommu=off` since 2026-09-11 (13-16 % prefill, measured upstream),
  what it cost (the NPU endpoint), what was deliberately not copied from
  the reference machine, and how to revert
- [[kingjones-strix-rocmfp4]] — PRIOR frank-large runbook and rollback
  target: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF, 177B params
  (56B MoE + 51.2B PLE), single 113.47 GiB GGUF, on the card's pinned
  kingjones30/ROCmFPX fork (2026-08-30 → 2026-09-11)
- [[ubatch-benchmark-large]] — active-model llama-bench ubatch sweep plan and results
- [[frank-medium-batch-ubatch]] — batch/ubatch sweep on the 2x R9700 layer split
  (2026-09-16): the ubatch optimum at 512, why the logical batch is free, and
  the split rebalance that followed it
- [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure
- [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B embedded-MTP model
  on Dankclaw (retired)
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on
  Dankserver (frank-small, retired 2026-09-02)
- [[ikllama-ornith-dankserver]] — the retired ik_llama.cpp runtime for
  frank-small (2026-08-26): q8_0 + Hadamard KV cache,
  split-mode none (single-GPU), mandatory FA_ALL_QUANTS build flags
- [[kingjones-strix-mtp]] — the former Q6 AGENT runbook (restored 2026-09-01
  after Q8_0 failure, then superseded by the STRIX deployment on 2026-08-27)
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime
  on Dankclaw (superseded 2026-08-24 by the kingjones ROCmFP4 model)
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment
  (deployed and reverted 2026-08-22)
- [[Dankserver System Specs]] — the 2x R9700 box (frank-medium)
- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large on the
  Kairic Edge runtime since 2026-09-21, frank-small on the RTX 3090)
- [[frank-npu]] — RETIRED 2026-09-11: the FLM NPU sight+sound endpoint
  on Dankclaw (Gemma4 E4B vision+audio pack, XDNA2, port 7771), kept as
  the retirement record — the IOMMU/SVA err=-19 root cause and where
  every dependent route moved

## CIRU Reference (Dankclaw)

- [[ciru-activefpx-promptforge]] — the RETIRED v3 model, ROCm 7.15 toolchain,
  and build history (on Dankclaw; the slot now serves the kingjones
  ROCmFP4 model, see [[kingjones-strix-mtp]])
- [[kairic-edge]] — Kairic Edge IU4 runtime: build runbook, binary
  identity, and experiment summary (retired 2026-08-22)

## DFlash2 Reference (archived)

- [[dflash2-large-vulkan]] — the retired DFlash2 Vulkan deployment
  (2026-08-21)

## vLLM Reference (historical)

- [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism
- [[VLLM Index]] — quick reference with commands, endpoints, and parameter table
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware
  notes and sizing (this architecture family is what the retired frank-small
  served in the Huihui abliterated FP8 variant — see
  [[freetoken-frank-small]]; the Ornith-1.5-35B-A3B CRACK checkpoint
  that preceded it is archived at [[dealign-ornith-crack]])

## Memory Management

- [[RAM CPU Offloading in VLLM]] — KV cache swap between GPU VRAM and system RAM
- [[Hybrid CPU GPU Processing Dankserver]] — NVIDIA GPU compute with system RAM offload
- [[Shared Memory Architecture Dankclaw]] — AMD unified memory and ROCm configuration

## Retired Topology (2026-08-17 redo)

- [[Qwen3.8 PP Deployment]] — the retired pipeline-parallel head + worker
- [[qwen3.8-spec-matrix]] — Qwen3.8-27B spec history (superseded by FastMTP)
- [[FastFlowLM NPU Inference]] — the retired FastFlowLM NPU endpoint
- [[frank-large-benchmark]] — backend/spec benchmark behind the old head
- [[dankserver-vram-growth]] — VRAM/RAM growth behind the retired cuBLAS crash
- [[DFlash DSpark Speculative Decoding]] — retired DFlash/DSpark drafting
- [[Quantization by Hardware Architecture]] — which quant format fits which GPU/CPU

## Pipeline Parallelism (archive)

- [[Pipeline Parallelism/index]] — index of multi-node inference research. The PP
  pair is retired. These pages are research history.

## Runtime Research (llama.cpp family)

- [[Llama Cpp Runtimes]] — mainline vs charlie12345/ROCmFPX vs ik_llama.cpp: positioning, quantization, server fit
- [[Llama Cpp Code Diff]] — code-level quantization diff of the three runtimes + server recommendations
- [[SGLang]] — the structured-generation serving engine; when it fits Dankserver (2x R9700) vs Dankclaw (Strix Halo)

## Related

- [Project knowledge base](../index.md) — the Podman app stacks on the same
  two hosts. OpenWebUI (Dankclaw) consumes the on-host `frank-large`
  endpoint at `:7770`. The Gitea instance on Dankclaw hosts this repository.
- [Deployment Flow](../Architecture/Deployment%20Flow.md) — the service-scoped
  git-based deploy with inference isolation