Frank
803398051a
Asked whether Vulkan on both devices unlocks the split. It does not: the per-tensor design aborts byte-identically, because the gate is not the driver mix but `--override-tensor` — `has_tensor_overrides()` is a required term of the scheduler's `pipeline_parallel` flag (llama-context.cpp:433, the only place in the tree that reads it), and that flag sets `n_copies` in `ggml_backend_sched_new`. With `n_copies == 1` a pre-allocated weight on a non-primary device has no backend and the graph aborts. Chasing it exposed a harness artifact that invalidated the earlier conclusion: llama-bench splits a device LIST with `/` and SWEEP values with `,`, so `-dev CUDA0,Vulkan1` was two single-device runs all along (and `-ts 0.85,0.15` two sweep values, i.e. device 0 takes everything — which is where the bogus OOM came from). With the correct syntax two-device splits work in both flavours: pure Vulkan -dev Vulkan1/Vulkan0 -ts 0.85/0.15 -> 27.55 tok/s @64k (12.8 GiB on the 3090) cross-vendor -dev CUDA0/Vulkan1 -ts 0.15/0.85 -> 27.23 tok/s @64k, pp512 268.5 (15.0 GiB) iGPU alone 25.77 tok/s @64k, pp512 231.8 +7 % decode, +16 % prefill — real but not a regime change, and it costs the 3090 ~15 GiB, which this fleet spends on its only vision endpoint. The design's premise still fails: whole layers carry their experts (1.62 GiB each), the 12 attention layers are spread every 4th, and the 32.8 GiB PLE must stay on the main device, so KV cannot be consolidated on the 3090. Also records the evening's incident (a dead rootless podman pause process took all container management down; `podman system migrate` panics in 5.7.0; reboot repaired it) and the netavark host-address lesson from the hermes 502.
194 lines
10 KiB
Markdown
194 lines
10 KiB
Markdown
# LLM Fleet — Knowledge Base
|
||
|
||
Documentation for the LLM inference fleet on the two hosts (2026-09-16):
|
||
**mainline llama.cpp on Dankserver** serving frank-medium as one layer-split
|
||
process across both Radeon AI PRO R9700 cards (ajgazin Swift-Qwen3.8-27B-
|
||
Uncensored-Dynamic-MTP UD-Q8_K_XL + the z-lab DFlash2 drafter), and TWO
|
||
endpoints on Dankclaw — halogen-flash-server (frank-large, Qwen3.8-Flash-Next
|
||
w4b on the gfx1151 iGPU, port 7770) and mainline llama.cpp (frank-small,
|
||
ISTA-DASLab Qwen3.8-27B-GSQ-RCO IQ3_S with MTP and vision on the RTX 3090, port
|
||
7771).
|
||
|
||
> 2026-08-22: the former `vllmScripts` repository is merged into the
|
||
> serverAppConfigs repository. This knowledge base moved from the old repo's
|
||
> `serverAppConfigs-docs/` root to `serverAppConfigs-docs/LLM/`. The fleet scripts and units now live in the
|
||
> repo's `scripts/` directory.
|
||
|
||
## Deployment Topology
|
||
|
||
```mermaid
|
||
graph LR
|
||
subgraph Dankserver[AMD Dankserver - Ubuntu 26.04]
|
||
A[2x Radeon AI PRO R9700
|
||
gfx1201, 32 GiB each]
|
||
B2[llama.cpp, layer split
|
||
frank-medium, port 7770]
|
||
C[Swift-Qwen3.8-27B-Uncensored UD-Q8_K_XL
|
||
ONE process on both cards, 262144 ctx / 1 slot, DFlash2]
|
||
A --> B2 --> C
|
||
end
|
||
|
||
subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
|
||
E2[RTX 3090 (PCIe x4 dock) +
|
||
gfx1151 iGPU 64 GiB unified memory]
|
||
E[halogen-flash-server 0.5.8
|
||
frank-large, port 7770]
|
||
F[peonist-ai Qwen3.8-Flash-Next w4b
|
||
2 x 200192 ctx over one 400384-position pool]
|
||
G[mainline llama.cpp 37b53fd
|
||
frank-small, port 7771]
|
||
H[Qwen3.8-27B GSQ-RCO IQ3_S
|
||
131072 ctx / 1 slot, MTP + vision]
|
||
E2 --> E --> F
|
||
E2 --> G --> H
|
||
end
|
||
|
||
CLI[LLM Clients / API] -->|HTTP :7770| B2
|
||
CLI -->|HTTP :7770| E
|
||
CLI -->|HTTP :7771| G
|
||
|
||
## Servers
|
||
|
||
| Server | GPU | VRAM / Memory | OS |
|
||
| --------------------------------------- | ----------------------------- | -------------------- | ------------ |
|
||
| [[Dankserver System Specs\|Dankserver]] | 2x AMD Radeon AI PRO R9700 | 2x 32 GB VRAM, 60 GB RAM | Ubuntu 26.04 |
|
||
| [[Dankclaw System Specs\|Dankclaw]] | AMD Radeon 8060S (Strix Halo) | gfx1151 unified memory | Ubuntu 26.04 |
|
||
|
||
## Active Deployment
|
||
|
||
- [[frank-medium-llamacpp-dflash2]] — the ACTIVE frank-medium runbook since
|
||
2026-09-16: mainline llama.cpp serving
|
||
ajgazin/Swift-Qwen3.8-27B-Uncensored-Dynamic-MTP-GGUF at UD-Q8_K_XL with the
|
||
z-lab DFlash2 Q8_0 drafter on Dankserver's 2x Radeon AI PRO R9700, as ONE
|
||
layer-split process behind :7770 (tensor parallelism is impossible on this
|
||
host — the second card cannot dispatch an RCCL collective)
|
||
- [[frank-medium-radiance-mxfp4]] — RETIRED 2026-09-16, one day after it was
|
||
deployed: the vllm-radiance MXFP4 stack (stilldeadcode/vllm-radiance:0.9.3 +
|
||
ggz14/radiance-vllm-mxfp4 patches) serving
|
||
just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq as two single-GPU replicas
|
||
behind a round-robin proxy; kept as the record of the deployment it
|
||
succeeded, plus the PCIe-atomics evidence that rules out TP=2 on this host
|
||
- [[frank-medium-rocmfpx]] — PRIOR frank-medium deployment (llama.cpp era,
|
||
2026-09-15): charlie12345/ROCmFPX serving
|
||
kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the
|
||
runtime it replaced
|
||
- [[freetoken-frank-small]] — RETIRED 2026-09-15, when the RTX 3090 moved to
|
||
frank-large: the FreeToken v0.1.2 frank-small runbook (batsclamp
|
||
Huihui-Qwen3.6-35B-A3B-abliterated-FP8), kept as the retirement record. The
|
||
NAME was reused on 2026-09-16 by a different service on Dankclaw — see
|
||
[[frank-small-qwen38-gsq-rco]]
|
||
- [[freetoken-benchmark-sweep]] — KV pool vs decode speed sweep
|
||
(bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
|
||
- [[deployment]] — deploy and update process for both servers
|
||
- [[frank-redo-2026-08-17]] — the 2026-08-17 topology change
|
||
- [[halogen-flash-server]] — ACTIVE frank-large runbook again since 2026-09-16
|
||
(rootless Quadlet, port 7770): peonist-ai halogen-flash-server 0.5.8 serving
|
||
peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn + quality
|
||
overlay) on the gfx1151 iGPU — two 200k conversations over one
|
||
400,384-position pool, deployed flags, memory model, client contract,
|
||
troubleshooting
|
||
- [[frank-small-qwen38-gsq-rco]] — ACTIVE frank-small runbook since 2026-09-16
|
||
(root systemd unit, port 7771): mainline llama.cpp serving
|
||
ISTA-DASLab/Qwen3.8-27B-GSQ-RCO IQ3_S on the RTX 3090, with the model's own
|
||
Multi-Token Prediction head for speculative decoding and the BF16 projector
|
||
for the fleet's image path
|
||
- [[x4-pair-experiment]] — MEASURED 2026-09-16: what mainline llama.cpp can and
|
||
cannot do with the 3090 + Strix Halo pair, over their only link (the 3090's
|
||
Gen4 x4). The tensor inventory that decides any split, the single flag (`-ot`)
|
||
that kills per-tensor placement across devices, the two-device layer splits
|
||
that work (+7 % decode at 64k, cross-vendor included), the harness-parsing trap
|
||
that invalidated the first round, and the iGPU-only result that matches
|
||
halogen's deployed decode rate — at a memory cost the co-tenants cannot pay
|
||
- [[frank-large-ik-uncensored]] — RETIRED 2026-09-16 (rollback target): the
|
||
ik_llama.cpp runbook (root systemd unit, 2026-09-15 → 09-16) serving
|
||
mradermacher/Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090
|
||
(dense/attention/KV) + LPDDR (experts/PLE) hybrid; its measured envelope and
|
||
decode root-cause chain are the record of why the fleet went back to halogen
|
||
- [[dankclaw-iommu-translation-off]] — why Dankclaw boots with
|
||
`amd_iommu=off` since 2026-09-11 (13-16 % prefill, measured upstream),
|
||
what it cost (the NPU endpoint), what was deliberately not copied from
|
||
the reference machine, and how to revert
|
||
- [[kingjones-strix-rocmfp4]] — PRIOR frank-large runbook and rollback
|
||
target: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF, 177B params
|
||
(56B MoE + 51.2B PLE), single 113.47 GiB GGUF, on the card's pinned
|
||
kingjones30/ROCmFPX fork (2026-08-30 → 2026-09-11)
|
||
- [[ubatch-benchmark-large]] — active-model llama-bench ubatch sweep plan and results
|
||
- [[kingjones-q8agent-rocmfpx]] — the retired Q8_0-AGENT investigation and `set_rows` failure
|
||
- [[eemin-rocmi4-mtp]] — the former caf-eemin Qwen3.8-27B embedded-MTP model
|
||
on Dankclaw (retired)
|
||
- [[dealign-ornith-crack]] — the retired Ornith-1.5-35B-A3B CRACK model on
|
||
Dankserver (frank-small, retired 2026-09-02)
|
||
- [[ikllama-ornith-dankserver]] — the retired ik_llama.cpp runtime for
|
||
frank-small (2026-08-26): q8_0 + Hadamard KV cache,
|
||
split-mode none (single-GPU), mandatory FA_ALL_QUANTS build flags
|
||
- [[kingjones-strix-mtp]] — the former Q6 AGENT runbook (restored 2026-09-01
|
||
after Q8_0 failure, then superseded by the STRIX deployment on 2026-08-27)
|
||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 CIRU PromptForge runtime
|
||
on Dankclaw (superseded 2026-08-24 by the kingjones ROCmFP4 model)
|
||
- [[kairic-edge]] — the retired Kairic Edge IU4 experiment
|
||
(deployed and reverted 2026-08-22)
|
||
- [[Dankserver System Specs]] — the 2x R9700 box (frank-medium)
|
||
- [[Dankclaw System Specs]] — the Strix Halo gfx1151 box (frank-large on
|
||
halogen-flash-server since 2026-09-16, frank-small on the RTX 3090)
|
||
- [[frank-npu]] — RETIRED 2026-09-11: the FLM NPU sight+sound endpoint
|
||
on Dankclaw (Gemma4 E4B vision+audio pack, XDNA2, port 7771), kept as
|
||
the retirement record — the IOMMU/SVA err=-19 root cause and where
|
||
every dependent route moved
|
||
|
||
## CIRU Reference (Dankclaw)
|
||
|
||
- [[ciru-activefpx-promptforge]] — the RETIRED v3 model, ROCm 7.15 toolchain,
|
||
and build history (on Dankclaw; the slot now serves the kingjones
|
||
ROCmFP4 model, see [[kingjones-strix-mtp]])
|
||
- [[kairic-edge]] — Kairic Edge IU4 runtime: build runbook, binary
|
||
identity, and experiment summary (retired 2026-08-22)
|
||
|
||
## DFlash2 Reference (archived)
|
||
|
||
- [[dflash2-large-vulkan]] — the retired DFlash2 Vulkan deployment
|
||
(2026-08-21)
|
||
|
||
## vLLM Reference (historical)
|
||
|
||
- [[VLLM Overview]] — installation, CLI, API, architecture, quantization, parallelism
|
||
- [[VLLM Index]] — quick reference with commands, endpoints, and parameter table
|
||
- [[Running Qwen3.6 35B A3B FP8 on Dankserver]] — Qwen3.6-35B-A3B hardware
|
||
notes and sizing (this architecture family is what the retired frank-small
|
||
served in the Huihui abliterated FP8 variant — see
|
||
[[freetoken-frank-small]]; the Ornith-1.5-35B-A3B CRACK checkpoint
|
||
that preceded it is archived at [[dealign-ornith-crack]])
|
||
|
||
## Memory Management
|
||
|
||
- [[RAM CPU Offloading in VLLM]] — KV cache swap between GPU VRAM and system RAM
|
||
- [[Hybrid CPU GPU Processing Dankserver]] — NVIDIA GPU compute with system RAM offload
|
||
- [[Shared Memory Architecture Dankclaw]] — AMD unified memory and ROCm configuration
|
||
|
||
## Retired Topology (2026-08-17 redo)
|
||
|
||
- [[Qwen3.8 PP Deployment]] — the retired pipeline-parallel head + worker
|
||
- [[qwen3.8-spec-matrix]] — Qwen3.8-27B spec history (superseded by FastMTP)
|
||
- [[FastFlowLM NPU Inference]] — the retired FastFlowLM NPU endpoint
|
||
- [[frank-large-benchmark]] — backend/spec benchmark behind the old head
|
||
- [[dankserver-vram-growth]] — VRAM/RAM growth behind the retired cuBLAS crash
|
||
- [[DFlash DSpark Speculative Decoding]] — retired DFlash/DSpark drafting
|
||
- [[Quantization by Hardware Architecture]] — which quant format fits which GPU/CPU
|
||
|
||
## Pipeline Parallelism (archive)
|
||
|
||
- [[Pipeline Parallelism/index]] — index of multi-node inference research. The PP
|
||
pair is retired. These pages are research history.
|
||
|
||
## Runtime Research (llama.cpp family)
|
||
|
||
- [[Llama Cpp Runtimes]] — mainline vs charlie12345/ROCmFPX vs ik_llama.cpp: positioning, quantization, server fit
|
||
- [[Llama Cpp Code Diff]] — code-level quantization diff of the three runtimes + server recommendations
|
||
- [[SGLang]] — the structured-generation serving engine; when it fits Dankserver (2x R9700) vs Dankclaw (Strix Halo)
|
||
|
||
## Related
|
||
|
||
- [Project knowledge base](../index.md) — the Podman app stacks on the same
|
||
two hosts. OpenWebUI (Dankclaw) consumes the on-host `frank-large`
|
||
endpoint at `:7770`. The Gitea instance on Dankclaw hosts this repository.
|
||
- [Deployment Flow](../Architecture/Deployment%20Flow.md) — the service-scoped
|
||
git-based deploy with inference isolation |