serverAppConfigs/llm/docs/Pipeline Parallelism/Mixed-Vendor Experiments.md

---

type: reference title: Mixed-Vendor Experiments description: Real-world NVIDIA + AMD Strix Halo multi-node experiments: what works (llama.cpp RPC, Vulkan split), what breaks (mixed CUDA+ROCm vLLM), measured tok/s. tags:

• pipeline-parallelism
• research timestamp: 2026-08-14

---

Research Brief: NVIDIA-Desktop + Strix-Halo Multi-Node LLM Inference

Summary

Yes, mixed NVIDIA-ROCm multi-node LLM inference over RPC has been published and is fully functional. The closest real experiments to an RTX-desktop + Strix-Halo-NUC pairing are (1) a planned 4× RTX 3090 + Strix Halo 128GB via 25GbE (r/LocalLLaMA, Apr 2026), (2) a Tesla V100 + AMD R9700×2 via Vulkan layer-split on one box (grosan.co.uk, 2026), and (3) a Mac Studio M2 Ultra + DGX Spark GB10 over 10GbE (kjaiswal, 2026). Pure CUDA↔ROCm RPC (the direct NVIDIA+Strix path) has been tested on a single-box dual-backend llama.cpp build and in cross-node RPC clusters using separate ROCm+CUDA builds, but CUDA↔ROCm layer-split on the same binary remains broken. Vulkan is the universal backend that works across vendors.

Verdict: The NVIDIA-desktop + Strix-Halo multi-node path via llama.cpp RPC is viable today. Single-binary same-machine ROCm+CUDA layer-split has a known bug (#11506). Cross-node RPC works with separate ROCm and CUDA builds on each machine. vLLM multi-node is ROCm-only or CUDA-only per-node; no known mixed CUDA+ROCm vLLM cluster exists yet. Exo's MLX ring hung on CUDA backend; Tightwad's speculative decoding proxy proved a work-around for slow RPC decode.

---

Findings

1. Published Experiments: NVIDIA + AMD Strix Halo Nodes

1a. Planned 4×RTX 3090 + Strix Halo 128GB cluster over 25GbE User on r/LocalLLaMA (Apr 2026) is building a llama.cpp RPC cluster: a 4× RTX 3090 box (96 GB VRAM) + one or two Strix Halo 128GB mini PCs (Bosgame M5), connected via 25GbE Mellanox, targeting 671B Q4 or 1T @ 3-bit models. Post was seeking community validation before purchasing. Source

1b. Mac Studio M2 Ultra (Metal) + DGX Spark GB10 (CUDA) over 10GbE kjaiswal published full benchmarks of splitting models across Apple Metal and NVIDIA CUDA nodes:

• Qwen2.5-7B Q4_K_M (4.4 GB): Local Metal 92 tok/s vs RPC Metal+CUDA 53 tok/s decode, but 4.2× faster prefill (318 vs 76 tok/s). The 7B fits both machines — RPC trades decode speed for prefill speed.
• Qwen2.5-72B Q4_K_M (44.2 GB): Local Metal 11 tok/s vs RPC Metal+CUDA 6 tok/s decode (~47% penalty). Split: 31 GB Metal + 14 GB CUDA.
• Projected for models needing both: MiniMax M2.5 (138 GB, ~5-8 tok/s), Qwen3-235B (132 GB, ~5-8 tok/s).
• Conclusion: RPC's value is capacity (100 GB+ models), not speed. Decode is 2× slower over RPC due to per-token network round-trips. Source

1c. Tesla V100 + AMD R9700×2 via Vulkan layer-split (single box) grosan.co.uk documented running Qwen3.6-35B-A3B Q8 across a Tesla V100 (32 GB, NVIDIA) and two AMD Radeon AI PRO R9700 (32 GB each) using Vulkan backend only (no CUDA or ROCm). Layer-split across three GPUs from two vendors. Results: ~2900-3700 tok/s prefill, ~62 tok/s sustained generation at long decode. Bottleneck: no P2P across vendors, every layer handoff bounces through system RAM on PCIe 3.0. Source

2. Strix Halo in Distributed Inference

2a. Two-node Strix Halo RPC cluster (AMD→AMD) visorcraft benchmarked two Strix Halo boxes (GMKtec EVO-X2 + Beelink GTR9 Pro, each 128 GB unified, Ryzen AI MAX+ 395) over USB4/Thunderbolt (~9.4 Gbps):

• MiniMax-M2.5-REAP-139B-Q8: 15.35 tok/s decode, RPC split 1.2/0.8
• Qwen3.5-397B-A17B-UD-Q4_K_XL: 11.76 tok/s decode (llama-bench), 12.6 tok/s (llama-server with -dio)
• Single-host single-node baseline: MiniMax-M2.5 Q3_K_M at 32.8 tok/s, Qwen3 30B-A3B at 86.1 tok/s
• Key caveat: RPC serving requires -dio flag on large models. Source

2b. AMD official: 4×Framework Desktop (Strix Halo) cluster AMD published a walkthrough running Kimi K2.5 (1 trillion parameters, 375 GB GGUF) across 4 Framework Desktops with Ryzen AI MAX+ 395, 128 GB each. Interconnect: 5 Gbps Ethernet. Flash Attention enabled via rocWMMA. Benchmarks at 8192 sequence length: 8.30 tok/s decode with FA enabled vs 3.46 tok/s without (batch/ubatch=4096). Context: 32768 tokens. Source

2c. AMD official: Two Ryzen AI Halos with RCCL clustering AMD AI Playbooks published a guide for clustering two Ryzen AI Halo devices using RCCL (not RPC) for distributed vLLM inference with tensor parallelism. Source

2d. vLLM on Strix Halo: dual-TP=2 over USB4 ayysasha benchmarked MiniMax-M2 (MoE, AWQ INT4) on two Strix Halos via vLLM tensor parallelism (TP=2) over USB4:

• Optimized MoE kernel: decode at 14.84 tok/s (1k context), 9.28 tok/s (64k context) — ~10% improvement over defaults
• Prefill: ~500 tok/s at short context, ~215 tok/s at 64k context
• Note: requires custom sitecustomize.py shim for device name detection on gfx1151. Source

2e. Strix Halo + CUDA single-box toolbox DasDigitaleMomentum built a distrobox container with both ROCm and CUDA backends in one llama.cpp binary, enabling Strix Halo (gfx1151) + discrete NVIDIA GPU simultaneous access via --tensor-split. Also supports dual-backend ROCm+Vulkan. Source

3. Heterogeneous Node Splitting: CUDA + ROCm RPC

3a. llama.cpp builds with both GGML_CUDA and GGML_HIP Issue #11506 reports that llama.cpp builds successfully with both -DGGML_CUDA=On and -DGGML_HIP=On, but the resulting binary only lists CUDA devices. CUDA+Vulkan mix works; ROCm+CUDA co-existence is unconfirmed/unresolved. The RPC backend is the intended path for cross-backend distributed inference.

3b. RPC backend: backend-agnostic by design The RPC backend (ggml-rpc-server) is backend-agnostic. Each worker runs its own llama.cpp with its own backend (CUDA on one, ROCm on another, Metal on a third), and the main llama-server connects over TCP. From llama.cpp's perspective, each remote machine is just another device. Source

3c. Known issues with RPC across different backends:

• Version skew: Both machines must use the exact same llama.cpp build tag; mismatched builds hang at handshake.
• Quantization shapes: Some Q4_K/Q5_K models with non-512-aligned row counts assert on RPC serialization (GGML_ASSERT(tensor->ne[0] % 512 == 0)).
• Network speed matters: WiFi → 10× slowdown vs wired ethernet. Ethernet 1G works but is latency-bound for decode.
• No authentication: rpc-server bound to 0.0.0.0 exposes all memory to any network client.
• Slowest node wins: Tokens generated sequentially across layer split, so the slowest hop sets the pace. Source

3d. Tightwad mixed-vendor cluster youngharold/tightwad pools CUDA + ROCm + Metal + CPU into one OpenAI-compatible endpoint with speculative decoding:

• Measured: 1.86× speedup on Llama 3.3 70B across 4 GPUs over WiFi (3 tok/s raw RPC → 4.1 tok/s with spec decode)
• Speculative decoding: Qwen3-235B with Qwen3-1.7B draft model over RPC — overall acceptance rate 6.6%, highest in factual (14.5%) and reasoning (13.8%) categories. Spec decode made slow RPC models actually usable. Source
• Config example mixed-vendor-pool.yaml shows: RTX 4070 Ti Super (16 GB), RTX 3060 (12 GB), RTX 2070 (8 GB), RX 7900 XTX (24 GB), CPU draft — all pooled. Source

3e. Combo-Llama: RTX 4070 + MI50 + RTX 3050 laptop (3-card hybrid) xxDoman's project connects RTX 4070 (12 GB), AMD MI50 (32 GB), and RTX 3050 laptop (4 GB) via RPC over 1 Gbps Ethernet:

• Qwen3-VL-30B Q5_K_M: Clean start 51.49 tok/s on Ethernet (WiFi was 44.53 tok/s)
• 32k context: 17.81 tok/s, 64k context: 12.90 tok/s (stable to 65536 tokens)
• MI50 at 89% GPU load, RTX 4070 at 12% — load imbalance typical. Source

4. Adjacent Projects

4a. Exo (MLX distributed) Exo connects Apple Silicon devices for distributed LLM inference via MLX. Tried cross-vendor Metal+CUDA over 10GbE with DGX Spark + Mac Studio. Result: Failed — mx.distributed.init(backend="ring") hangs on CUDA backend (as of MLX 0.31.1). CUDA ring support from Apple is the blocker. Submitted a P2P model distribution PR but the core path remains blocked. Source

4b. Petals Distributed inference framework. Not yet tested on Strix Halo. Uses its own custom protocol over TCP. [General reference]

4c. llama.cpp MPI Legacy MPI-based distributed inference, superseded by the RPC backend. RPC is now the standard multi-node path. [llama.cpp docs]

4d. Poor man's cluster / ethernet experiments dev.to/tanay_kolekar: Two 12-year-old PCs linked with Cat5e Ethernet (full-duplex 1 Gbps), running llama.cpp RPC. Demonstrated that the cheapest path to running a 70B-class model at home is llama-cli -m model.gguf --rpc <worker_ip>:50052. Source

4e. Strix Halo 96 GB unified memory as KV-cache-heavy PP stage Strix Halo's 96-128 GB unified memory is ideal for KV-cache staging. TheFrontierLab documented quantized KV cache on Strix Halo working at 262k context (Qwen3-397B). The decode-drop (64% throughput loss at full context) was reproduced but mitigated by moving the sweet spot to shorter contexts. Source

4f. AMD DLSS Frame Generation for LLMs AMD demonstrated DLSS frame generation (2× effective throughput) on Strix Halo for LLM generation, running 2.4× faster generation via frame generation without accuracy loss. Source

4g. RDMA/RoCEv2 on llama.cpp RPC PR #20590 added native RDMA (RoCEv2) transport to llama.cpp RPC. Tested on two-node AMD Radeon 8060S (gfx1151) cluster with ConnectX-4/6 25GbE:

• Qwen3-Coder-Next 80B Q8_K_XL, layer split across two nodes
• TCP: 651.48 pp2048 / 30.19 tg256
• RDMA: 678.42 pp2048 / 32.16 tg256 (+4.1% / +6.5% improvement) Source

5. Measured Numbers Summary

Experiment	Hardware	Model	Network	Prefill (tok/s)	Decode (tok/s)	Context
Mac Studio+DGX Spark (Metal+CUDA)	M2 Ultra + GB10	Qwen2.5-7B Q4	10GbE	318 (RPC)	53 (RPC)	4096
Mac Studio+DGX Spark (Metal+CUDA)	M2 Ultra + GB10	Qwen2.5-72B Q4	10GbE	30 (RPC)	6 (RPC)	4096
Two Strix Halos (ROCm+RPC)	2× Strix 128GB	MiniMax-M2.5 Q8	USB4 9.4Gbps	332	15.35	—
Two Strix Halos (ROCm+RPC)	2× Strix 128GB	Qwen3.5-397B Q4	USB4 9.4Gbps	148	11.76	—
AMD 4-node Strix (ROCm+RPC)	4× Framework 128GB	Kimi K2.5 Q2	5GbE	—	8.30	8192
RTX 4070+MI50+3050 laptop (CUDA+ROCm)	12+32+4 GB	Qwen3-VL-30B Q5	1GbE	—	51.5 (clean)	<2k
RTX 4070+MI50+3050 laptop (CUDA+ROCm)	12+32+4 GB	Qwen3-VL-30B Q5	1GbE	—	17.8	32768
Tesla V100+R9700×2 (Vulkan)	32+32+32 GB	Qwen3.6-35B-A3B Q8	PCIe 3.0	2900-3700	62	262144
Tightwad spec decode	Mixed CUDA+ROCm	Llama 3.3 70B	WiFi	—	4.1 (spec) vs 2.2	—
Tightwad spec decode	Mixed CUDA+ROCm	Qwen3-235B	RPC pool	—	~6.6% accept rate	256 max

Key pattern: Decoding over RPC is 2-5× slower than local. Prefill benefits from more compute cores. Speculative decoding is the best way to make RPC decode usable. For models that fit one machine, local is always faster.

---

What Works Today

1. llama.cpp RPC: CUDA node + ROCm node — Build separate llama.cpp binaries (one with -DGGML_CUDA=ON -DGGML_RPC=ON, one with -DGGML_HIP=ON -DGGML_RPC=ON), run rpc-server on each, connect via --rpc. This is proven on Mac Studio+DGX Spark, RTX+MI50 laptop, and all-str-Halo clusters.

2. llama.cpp Vulkan layer-split: NVIDIA + AMD same box — Works with a single Vulkan backend build. Tested on Tesla V100 + R9700×2.

3. Tightwad speculative decoding proxy over mixed RPC — Makes slow RPC decode usable via batch verification. 1.86× measured speedup on 70B.

4. Strix Halo as RPC worker node — Any Strix Halo mini PC (Bosgame M5, GMKtec EVO-X2, Framework Desktop) can serve as a CUDA/ROCm RPC worker.

5. RDMA/RoCEv2 transport — PR #20590 merged. Provides 4-6% improvement over TCP on Strix Halo→Strix Halo.

What Breaks

1. Single-binary ROCm+CUDA layer-split — Issue #11506. Build compiles but only lists CUDA devices. Unresolved.

2. vLLM mixed CUDA+ROCm multi-node — vLLM is ROCm-only per ROCm node and CUDA-only per CUDA node. No built-in mechanism to span TP across CUDA→ROCm boundary. Works via Ray multi-node but both ends must be same backend.

3. Exo MLX ring on CUDA — mx.distributed.init() hangs on CUDA backend. Blocks cross-vendor Apple+NVIDIA clusters.

4. RPC version skew — Any llama.cpp tag mismatch between master and workers causes hangs. All nodes must use identical builds.

5. Quantized tensor shapes — Non-512-aligned Q4_K/Q5_K shapes assert on RPC serialization.

---

Closest Real Experiment to NVIDIA-desktop + Strix-Halo

r/LocalLLaMA post (Apr 2026): User plans 4× RTX 3090 box (96 GB VRAM, CUDA) + 1-2× Strix Halo 128GB mini PCs over 25GbE Mellanox for llama.cpp RPC, targeting 671B Q4 or 1T @ 3-bit. Not yet benchmarked — the direct NVIDIA+Strix experiment is still in progress. Source

Closest completed experiment: kjaiswal's Mac Studio (Metal) + DGX Spark (CUDA) over 10GbE, where the Mac Studio is the closest consumer equivalent to a Strix Halo desktop in terms of unified memory architecture. This proves the CUDA+unified-memory-node pattern. Source

---

Recommended Architecture for NVIDIA-desktop + Strix-Halo

┌──────────────────────────┐     Ethernet/RoCE      ┌──────────────────────────┐
│  NVIDIA Desktop Node     │  (10GbE or 25GbE)       │  Strix Halo Mini PC      │
│  RTX 3090/4090 (24 GB)   │◄──────────────────────►│  Ryzen AI MAX+ 395         │
│  llama.cpp CUDA + RPC    │                          │  Radeon 8060S (96 GB)      │
│  llama-server (head)     │                          │  llama.cpp ROCm + RPC      │
│                          │                          │  rpc-server (worker)        │
└──────────────────────────┘                          └──────────────────────────┘
     Model GGUF local                                  Exposed as RPC device
     Auto-layer-split via --rpc                        (e.g., -m 90000 for 90 GB)

• Strategy: Use Strix Halo as an RPC worker contributing 90+ GB of unified memory for model weights and KV cache. NVIDIA RTX node acts as head/primary with llama-server.
• Networking: 10GbE minimum for usable decode. 25GbE/RoCE for 4+ tok/s on 100B+ models.
• Backends: Separate builds per machine (CUDA on NVIDIA, ROCm on Strix). No single-binary cross-backend needed.
• Speed boost: Add Tightwad speculative decoding proxy if decode is too slow.
• Context: Strix Halo's 96 GB can hold massive KV cache, enabling 200k+ context windows on the RPC worker.

---

Sources

Kept:

• kjaiswal/llama-cpp-distributed-benchmarks — Metal+CUDA RPC benchmarks on Mac Studio+DGX Spark, real tok/s numbers, prefill vs decode analysis
• grosan.co.uk: Vulkan layer-split hybrid — Tesla V100 + R9700×2 Vulkan split, 2900-3700 pp tok/s, 62 tg tok/s
• visorcraft/strix-halo-llm-perf — Two Strix Halos over USB4 RPC, MiniMax-M2.5 Q8 at 15.35 tok/s, Qwen3.5-397B at 11.76 tok/s
• AMD Ryzen AI Max+ 4-node cluster — Official AMD Kimi K2.5 1T parameter run, 4× Strix, 8.30 tok/s with Flash Attention
• sharedllm.org: RPC manual — Definitive RPC guide, known issues, version skew, WiFi vs ethernet
• youngharold/tightwad — Mixed CUDA+ROCm+Metal cluster with spec decode, 1.86× speedup on 70B
• xxDoman/combo-llama-rpc-hybrid — RTX 4070 + MI50 + 3050 laptop hybrid, 51.5 tok/s clean start on 1GbE
• DasDigitaleMomentum/strix-halo-cuda-combined-toolbox — Single-container llama.cpp with ROCm+CUDA, Strix Halo+discrete NVIDIA
• ayysasha/Strix-halo-dual-optimized — vLLM TP=2 on dual Strix via USB4, MiniMax-M2 MoE kernels
• AMD RCCL clustering — Two Ryzen AI Halos with RCCL for vLLM
• llama.cpp RPC RDMA PR #20590 — RoCEv2 on two Strix 8060S, +4-6% improvement
• llama.cpp issue #11506 — Single-binary ROCm+CUDA bug report
• r/LocalLLaMA: 4×3090 + Strix Halo — Planned NVIDIA+Strix over 25GbE
• TheFrontierLab: Quantized KV cache on Strix — 262k context on Strix Halo, decode-drop analysis

Dropped:

• Airliners.net — Irrelevant airline results in Exa search
• exo-explore/exo — K-pop group results, not the distributed inference project
• vLLM ROCm device map — Confirms Strix Halo (gfx1151) support but no mixed-node experiment data
• vLLM issue #45751 — WideTP/WideEP on ROCm, useful but no mixed CUDA data

---

Gaps

1. Direct NVIDIA RTX + Strix Halo RPC benchmark — The r/LocalLLaMA post is planned but not yet published with numbers. Awaiting publication.
2. Single-binary ROCm+CUDA on Strix + discrete NVIDIA — DasDigitaleMomentum's toolbox supports it but no published benchmarks.
3. vLLM CUDA+ROCm mixed-node — No known real experiment. Both vLLM nodes use same backend.
4. RTX 4090 + Strix Halo via Thunderbolt — No published experiment. USB4 on Strix → Thunderbolt on NVIDIA laptop not tested.
5. Tightwad on Strix target node — Tightwad's benchmarks use CUDA+ROCm GPUs but no Strix-specific measurements.

Suggested next steps:

• Build the exact setup: RTX 4090 node (llama-server) + Strix Halo mini (rpc-server) over 10GbE.
• Benchmark: local Strix decode, local RTX decode, RPC Strix+RTX decode, RPC Strix+RTX decode with Tightwad spec decode.
• Test context scaling: 32k, 64k, 128k, 256k on the RPC path.
• Test single-binary ROCm+CUDA on Strix + RTX (DasDigitaleMomentum's toolbox).