serverAppConfigs/docs/Pipeline Parallelism/Expected Performance.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	Expected Performance	Estimated tokens per second for a 25-40B model split across two nodes over 1/2.5/10 GbE, with real benchmark anchors.	pipeline-parallelism performance estimate	2026-08-14

Expected Performance

This page estimates the speed of a 25-40B model split across two machines over ethernet. It uses the bandwidth math in Ethernet Bandwidth Math and the measured numbers in Mixed-Vendor Experiments.

The Bandwidth Ceiling Is Not the Limit

A pipeline boundary moves only the hidden state. For a 27B model (hidden 5120), that is 10240 bytes per token per boundary in fp16. Even at 64 tokens per second with batch 64, the traffic is about 262 Mbit/s. 1 GbE can carry it. The real limits are latency, RPC overhead, and the slowest node.

The Petals paper measured this directly. Dropping from 1 Gbit/s to 100 Mbit/s changed throughput by 3 percent. Raising round-trip latency from 5 ms to 100 ms dropped throughput by 28 percent. Latency matters more than bandwidth for pipeline inference.

Real Benchmark Anchors

Experiment	Model	Network	Decode (t/s)
Mac Studio + DGX Spark (Metal+CUDA)	Qwen2.5-72B Q4	10 GbE	6
Mac Studio + DGX Spark (Metal+CUDA)	Qwen2.5-7B Q4	10 GbE	53
RTX 4070 + MI50 + 3050 (CUDA+ROCm)	Qwen3-VL-30B Q5	1 GbE	51.5 clean, 17.8 at 32k ctx
Two Strix Halos (ROCm RPC)	MiniMax-M2.5-139B Q8	USB4	15.35
AMD four-node Strix Halo (5 GbE)	Kimi K2.5 (1T)	5 GbE	8.30
Tesla V100 + R9700x2 (Vulkan, one box)	Qwen3.6-35B-A3B Q8	PCIe	62

Pattern: RPC decode runs about 2-5 times slower than the best single node. Prefill benefits from more compute across nodes. The 30B-class model over 1 GbE with warm cache reached 51.5 t/s on a short context and 17.8 t/s at 32k context.

Estimate for Qwen3.6-27B Q8, Two Nodes

Scenario: RTX 3090 (24 GB VRAM, NVIDIA desktop) plus Strix Halo NUC (Radeon 8060S, 30 GB physical RAM). The model is the dense 27B at Q8 (29-32 GB). Each node holds part of the weights and KV cache.

Link	Decode (t/s)	Prefill (t/s)	Notes
1 GbE	8-15	200-500	Latency-bound; per-token RPC round trip
2.5 GbE	10-18	250-600	Some headroom; latency still dominant
10 GbE	12-20	300-800	Closest to the published 10 GbE runs

The dense 27B decodes slower per token than the MoE 30B A3B class. Expect the lower half of the ranges at full context.

With Speculative Decoding

Speculative decoding (MTP, DFlash, or ngram-mod) raises effective decode. Tightwad measured 1.86x on a slow RPC pool. Estimate the effective rate at 1.4-1.8x the raw RPC decode:

Link	Raw decode (t/s)	With spec (t/s)
1 GbE	8-15	12-24
2.5 GbE	10-18	15-28
10 GbE	12-20	18-32

Context That Fits

The two real machines give about 51 GiB of usable memory combined (24 GiB VRAM plus about 27 GiB on the Strix Halo, which has 30 GiB physical RAM; the 96 GiB driver ceiling is not physical). A 29-32 GB model leaves about 20-22 GiB for KV cache. The Qwen3.6-27B KV cache costs about 85-100 KiB per token with q8_0 K and f16 V (measured on the large service at 602100 tokens = about 50 GB). That budget gives a total context pool of roughly 200k-260k tokens across both nodes.

Caveats

• Every node must run the exact same llama.cpp build. Version skew hangs the handshake.
• The slowest node sets the pace for every token.
• Model load crosses the network once per start. Expect minutes, not seconds, for the split load.
• These are estimates. The exact combination of an RTX 3090 desktop and a Strix Halo NUC has no published benchmark yet.

See Also

• Ethernet Bandwidth Math
• Mixed-Vendor Experiments
• Sample Script Set