Files

T

Frank 4fca739bf8 frank-medium: deploy two single-GPU replicas — TP=2 is blocked by PCIe atomics

The radiance MXFP4 stack cannot run TP=2 on this host. RCCL's collective
kernels carry a hidden hostcall-buffer parameter, and ROCr refuses to
dispatch them on a device without PCIe atomics: the AQL dispatch fails with
hipErrorIllegalState ("the operation cannot be performed in the present
state"), which is what every two-rank collective hit. The second R9700 sits
behind the chipset bridges 05:00.0/06:09.0, which advertise
`AtomicOpsCap: Routing-` — a read-only capability, so no software can enable
atomic routing on that path and the driver's
pci_enable_atomic_ops_to_root() fails for that card. The first card, on CPU
root port 00:01.1, has atomics enabled.

Ruled out: kernel version (7.0.0-28/-30/-31 identical, after a full upgrade),
iommu=pt, P2P hardware (45.3 GB/s between the cards), the shipped librccl (a
from-source rebuild reproduces the failure at the same call path), vLLM and
pynccl (a bare two-rank torch.distributed allreduce fails the same way), all
NCCL transport knobs, and single-rank tests (NCCL short-circuits at
world_size=1, so one-card success proves nothing).

Deployment: three rootless Quadlets — frank-medium-a (:7771, card 1),
frank-medium-b (:7772, card 2), both TP=1 at 114688 context (262144 and
131072 do not fit one card's KV) with vLLM-profiled KV — and
frank-medium-proxy (:7770), a round-robin that preserves the fleet contract
of one endpoint and one served model name. frank-medium.container stays as
the TP=2 target for a host whose second card is atomic-capable; the deploy
handler does not install it and trashes any stale copy.

Measured through the proxy (vllm bench serve, random dataset, --ignore-eos):
conc 1 42.8 tok/s, conc 4 147.6 tok/s, conc 8 230.8 tok/s, 32k prefill at
~2.8k tok/s per card, TPOT p50 23.6-28.8 ms.

Also: healthcheck reads PORT so one probe serves all three units, the hermes
example and the harness models.yml drop to the real 114688 context, and the
runbook records the full evidence chain.

2026-09-16 03:47:11 -07:00

8.2 KiB

Raw Blame History

LLM Fleet — Knowledge Base

Documentation for the LLM inference fleet on the two hosts (2026-09-16): charlie12345/ROCmFPX llama.cpp on Dankserver (frank-medium, kingjones777 Qwen3.8-27B-ROCmFPX-Q8_0-AGENT on 2x Radeon AI PRO R9700) and ik_llama.cpp on Dankclaw (frank-large, mradermacher Qwen3.8-Flash-Next- Uncensored i1-Q4_K_M on the RTX 3090 + Strix Halo hybrid).

2026-08-22: the former vllmScripts repository is merged into the serverAppConfigs repository. This knowledge base moved from the old repo's serverAppConfigs-docs/ root to serverAppConfigs-docs/LLM/. The fleet scripts and units now live in the repo's scripts/ directory.

Deployment Topology

graph LR
    subgraph Dankserver[AMD Dankserver - Ubuntu 26.04]
        A[2x Radeon AI PRO R9700
gfx1201, 32 GiB each]
        B2[ROCmFPX llama.cpp
frank-medium, port 7770]
        C[Qwen3.8-27B-ROCmFPX-Q8_0-AGENT
layer split, 131k ctx / 1 slot, MTP]
        A --> B2 --> C
    end

    subgraph Dankclaw[AMD Strix Halo - Ubuntu 26.04]
        E2[RTX 3090 (PCIe x4 dock) +
gfx1151 iGPU 64 GiB unified memory]
        E[ik_llama.cpp master d85a1d7
frank-large, port 7770]
        F[mradermacher Qwen3.8-Flash-Next-Uncensored
i1-Q4_K_M GGUF, 262k ctx / 1 slot, vision]
        E2 --> E --> F
    end

    CLI[LLM Clients / API] -->|HTTP :7770| B2
    CLI -->|HTTP :7770| E

Servers

Server	GPU	VRAM / Memory	OS
Dankserver System Specs	2x AMD Radeon AI PRO R9700	2x 32 GB VRAM, 60 GB RAM	Ubuntu 26.04
Dankclaw System Specs	AMD Radeon 8060S (Strix Halo)	gfx1151 unified memory	Ubuntu 26.04

Active Deployment

frank-medium-radiance-mxfp4 — the ACTIVE frank-medium runbook: the vllm-radiance MXFP4 stack (stilldeadcode/vllm-radiance:0.9.3 + ggz14/radiance-vllm-mxfp4 patches) serving just1moremodel/Qwen3.8-27B-Uncensored-MXFP4-awq on Dankserver's 2x Radeon AI PRO R9700 as two single-GPU replicas behind a round-robin proxy, with the PCIe-atomics evidence that rules out TP=2 on this host (since 2026-09-15/16)
frank-medium-rocmfpx — PRIOR frank-medium deployment (llama.cpp era, 2026-09-15): charlie12345/ROCmFPX serving kingjones777/Qwen3.8-27B-ROCmFPX-Q8_0-AGENT-GGUF; kept as the record of the runtime it replaced
freetoken-frank-small — RETIRED 2026-09-15 with the RTX 3090: the FreeToken v0.1.2 frank-small runbook (batsclamp Huihui-Qwen3.6-35B-A3B- abliterated-FP8), kept as the retirement record
freetoken-benchmark-sweep — KV pool vs decode speed sweep (bs1/bs3/bs4/bs6, 8k–700k KV) on Dankserver RTX 3090
deployment — deploy and update process for both servers
frank-redo-2026-08-17 — the 2026-08-17 topology change
frank-large-ik-uncensored — ACTIVE frank-large runbook since 2026-09-15: ik_llama.cpp (root systemd unit) serving mradermacher/ Qwen3.8-Flash-Next-Uncensored-i1-Q4_K_M on the RTX 3090 (dense/attention/ KV) + LPDDR (experts/PLE) hybrid; measured performance envelope, the decode root-cause chain, tuning levers, and the co-tenancy traps
halogen-flash-server — RETIRED 2026-09-15 (rollback target): the peonist-ai halogen-flash-server 0.5.8 runbook serving peonist-ai/halogen-qwen3.8-flash-next (Qwen3.8-Flash-Next w4b .hgn + quality overlay), rootless Quadlet container on Dankclaw port 7770 — deployed flags, memory model, client contract, troubleshooting
dankclaw-iommu-translation-off — why Dankclaw boots with amd_iommu=off since 2026-09-11 (13-16 % prefill, measured upstream), what it cost (the NPU endpoint), what was deliberately not copied from the reference machine, and how to revert
kingjones-strix-rocmfp4 — PRIOR frank-large runbook and rollback target: kingjones777/Qwen3.8-Flash-Next-ROCmFP4-STRIX-GGUF, 177B params (56B MoE + 51.2B PLE), single 113.47 GiB GGUF, on the card's pinned kingjones30/ROCmFPX fork (2026-08-30 → 2026-09-11)
ubatch-benchmark-large — active-model llama-bench ubatch sweep plan and results
kingjones-q8agent-rocmfpx — the retired Q8_0-AGENT investigation and set_rows failure
eemin-rocmi4-mtp — the former caf-eemin Qwen3.8-27B embedded-MTP model on Dankclaw (retired)
dealign-ornith-crack — the retired Ornith-1.5-35B-A3B CRACK model on Dankserver (frank-small, retired 2026-09-02)
ikllama-ornith-dankserver — the retired ik_llama.cpp runtime for frank-small (2026-08-26): q8_0 + Hadamard KV cache, split-mode none (single-GPU), mandatory FA_ALL_QUANTS build flags
kingjones-strix-mtp — the former Q6 AGENT runbook (restored 2026-09-01 after Q8_0 failure, then superseded by the STRIX deployment on 2026-08-27)
ciru-activefpx-promptforge — the RETIRED v3 CIRU PromptForge runtime on Dankclaw (superseded 2026-08-24 by the kingjones ROCmFP4 model)
kairic-edge — the retired Kairic Edge IU4 experiment (deployed and reverted 2026-08-22)
Dankserver System Specs — the 2x R9700 box (frank-medium)
Dankclaw System Specs — the Strix Halo gfx1151 box (frank-large, on halogen-flash-server since 2026-09-11)
frank-npu — RETIRED 2026-09-11: the FLM NPU sight+sound endpoint on Dankclaw (Gemma4 E4B vision+audio pack, XDNA2, port 7771), kept as the retirement record — the IOMMU/SVA err=-19 root cause and where every dependent route moved

CIRU Reference (Dankclaw)

ciru-activefpx-promptforge — the RETIRED v3 model, ROCm 7.15 toolchain, and build history (on Dankclaw; the slot now serves the kingjones ROCmFP4 model, see kingjones-strix-mtp)
kairic-edge — Kairic Edge IU4 runtime: build runbook, binary identity, and experiment summary (retired 2026-08-22)

DFlash2 Reference (archived)

dflash2-large-vulkan — the retired DFlash2 Vulkan deployment (2026-08-21)

vLLM Reference (historical)

VLLM Overview — installation, CLI, API, architecture, quantization, parallelism
VLLM Index — quick reference with commands, endpoints, and parameter table
Running Qwen3.6 35B A3B FP8 on Dankserver — Qwen3.6-35B-A3B hardware notes and sizing (this architecture family is what the retired frank-small served in the Huihui abliterated FP8 variant — see freetoken-frank-small; the Ornith-1.5-35B-A3B CRACK checkpoint that preceded it is archived at dealign-ornith-crack)

Memory Management

RAM CPU Offloading in VLLM — KV cache swap between GPU VRAM and system RAM
Hybrid CPU GPU Processing Dankserver — NVIDIA GPU compute with system RAM offload
Shared Memory Architecture Dankclaw — AMD unified memory and ROCm configuration

Retired Topology (2026-08-17 redo)

Qwen3.8 PP Deployment — the retired pipeline-parallel head + worker
qwen3.8-spec-matrix — Qwen3.8-27B spec history (superseded by FastMTP)
FastFlowLM NPU Inference — the retired FastFlowLM NPU endpoint
frank-large-benchmark — backend/spec benchmark behind the old head
dankserver-vram-growth — VRAM/RAM growth behind the retired cuBLAS crash
DFlash DSpark Speculative Decoding — retired DFlash/DSpark drafting
Quantization by Hardware Architecture — which quant format fits which GPU/CPU

Pipeline Parallelism (archive)

Pipeline Parallelism/index — index of multi-node inference research. The PP pair is retired. These pages are research history.

Runtime Research (llama.cpp family)

Llama Cpp Runtimes — mainline vs charlie12345/ROCmFPX vs ik_llama.cpp: positioning, quantization, server fit
Llama Cpp Code Diff — code-level quantization diff of the three runtimes + server recommendations
SGLang — the structured-generation serving engine; when it fits Dankserver (2x R9700) vs Dankclaw (Strix Halo)

Project knowledge base — the Podman app stacks on the same two hosts. OpenWebUI (Dankclaw) consumes the on-host frank-large endpoint at :7770. The Gitea instance on Dankclaw hosts this repository.
Deployment Flow — the service-scoped git-based deploy with inference isolation

8.2 KiB Raw Blame History Unescape Escape