serverAppConfigs/docs/Pipeline Parallelism/Ethernet Bandwidth Math.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	Ethernet Bandwidth Math	Bytes-per-token math for pipeline vs tensor parallelism, feasibility tables for 1/2.5/10 GbE, and real multi-node benchmark numbers.	pipeline-parallelism research	2026-08-14

Bandwidth Analysis: Multi-Node LLM Inference over Ethernet

1. Bandwidth Math: Bytes per Token per Layer Boundary

1.1 Pipeline Parallelism (PP)

During autoregressive generation, each transformer layer produces a hidden state of shape (batch_size, hidden_size). At a pipeline stage boundary, this tensor must be communicated between nodes.

Forward pass only (inference):

bytes_per_boundary = batch_size × hidden_size × 2   [fp16 = 2 bytes]

Key detail: During generation (one token at a time), seq_len = 1 at each boundary crossing. The KV cache stays local. The hidden state is (batch, hidden_size) — independent of context length.

For a 27B dense model (e.g., Qwen3.6-27B):

• Hidden size: H = 4096 (standard for Llama/Qwen-style architectures)
• Layers: L = 32 (typical)
• Per-token, per-boundary: 4096 × 2 = 8,192 bytes = 8 KB
• With Nₚₚ stage boundaries: total = Nₚₚ × 8 KB/tok

For a 36B MoE model (e.g., Qwen3.6-35B-A3B):

• Hidden size: H = 6144 (common for this scale)
• Layers: L = 48 (typical)
• Per-token, per-boundary: 6144 × 2 = 12,288 bytes = 12 KB
• With Nₚₚ stage boundaries: total = Nₚₚ × 12 KB/tok

Full example — 27B model, 2 PP stages (1 boundary), batch=1:

1 boundary × 4096 × 2 bytes = 8,192 bytes/tok

Example — same model, batch=64:

1 boundary × 4096 × 64 × 2 bytes = 524,288 bytes = 512 KB/tok

Example — 36B model, 2 PP stages, batch=1:

1 boundary × 6144 × 2 bytes = 12,288 bytes = 12 KB/tok

1.2 Tensor Parallelism (TP) — for comparison

In tensor parallelism, each MLP and attention layer performs an all-reduce on the hidden state:

bytes_per_layer = batch_size × hidden_size × 2   [one all-reduce per layer]
total_bytes_per_tok = L × batch_size × hidden_size × 2   [L all-reduces total]

For 27B, H=4096, L=32, TP=2, batch=1:

32 layers × 4096 × 1 × 2 bytes = 262,144 bytes = 256 KB/tok
(16 KB per layer all-reduce × 32 layers)

For 36B, H=6144, L=48, TP=2, batch=1:

48 layers × 6144 × 1 × 2 bytes = 589,824 bytes = 576 KB/tok
(12 KB per layer all-reduce × 48 layers)

1.3 Scaling with batch size

Model	H	L	tok/s	Batch	PP boundary bytes/tok	TP total bytes/tok
27B	4096	32	10	1	8 KB	256 KB
27B	4096	32	50	1	8 KB	256 KB
27B	4096	32	100	1	8 KB	256 KB
27B	4096	32	10	16	128 KB	4,096 KB
27B	4096	32	50	16	128 KB	4,096 KB
27B	4096	32	100	16	128 KB	4,096 KB
27B	4096	32	10	64	512 KB	16,384 KB
27B	4096	32	50	64	512 KB	16,384 KB
27B	4096	32	100	64	512 KB	16,384 KB
36B	6144	48	10	1	12 KB	576 KB
36B	6144	48	50	1	12 KB	576 KB
36B	6144	48	100	1	12 KB	576 KB
36B	6144	48	10	16	192 KB	9,216 KB
36B	6144	48	50	16	192 KB	9,216 KB
36B	6144	48	100	16	192 KB	9,216 KB
36B	6144	48	10	64	768 KB	36,864 KB
36B	6144	48	50	64	768 KB	36,864 KB
36B	6144	48	100	64	768 KB	36,864 KB

---

2. Required Bandwidth vs. Ethernet Capabilities

2.1 Ethernet link capacities

Link	Raw Rate	Usable TCP (approx.)	MB/s usable
1 GbE	1,000 Mbit/s	~940 Mbit/s	~118 MB/s
2.5 GbE	2,500 Mbit/s	~2,350 Mbit/s	~294 MB/s
10 GbE	10,000 Mbit/s	~9,400 Mbit/s	~1,175 MB/s

2.2 Required bandwidth (Mbps) by model, batch, and throughput

27B model (H=4096), 1 PP boundary:

tok/s	Batch=1	Batch=16	Batch=64
10	0.13 Mbit/s	2.05 Mbit/s	8.19 Mbit/s
50	0.65 Mbit/s	10.24 Mbit/s	40.96 Mbit/s
100	1.31 Mbit/s	20.48 Mbit/s	81.92 Mbit/s

36B model (H=6144), 1 PP boundary:

tok/s	Batch=1	Batch=16	Batch=64
10	0.20 Mbit/s	3.07 Mbit/s	12.29 Mbit/s
50	0.98 Mbit/s	15.36 Mbit/s	61.44 Mbit/s
100	1.97 Mbit/s	30.72 Mbit/s	122.88 Mbit/s

27B model, TP (L=32 layers all-reduce):

tok/s	Batch=1	Batch=16	Batch=64
10	2.05 Mbit/s	32.77 Mbit/s	131.07 Mbit/s
50	10.24 Mbit/s	163.84 Mbit/s	655.36 Mbit/s
100	20.48 Mbit/s	327.68 Mbit/s	1,310.72 Mbit/s

2.3 Feasibility matrix

Green = feasible (< 50% of link capacity). Yellow = marginal (50–90%). Red = saturated (> 90%).

27B model — PP (1 boundary), batch=1:

	1 GbE (118 MB/s)	2.5 GbE (294 MB/s)	10 GbE (1175 MB/s)
10 tok/s	✅ 0.13 Mbit/s	✅ 0.13 Mbit/s	✅ 0.13 Mbit/s
50 tok/s	✅ 0.65 Mbit/s	✅ 0.65 Mbit/s	✅ 0.65 Mbit/s
100 tok/s	✅ 1.31 Mbit/s	✅ 1.31 Mbit/s	✅ 1.31 Mbit/s

27B model — PP (1 boundary), batch=16:

	1 GbE	2.5 GbE	10 GbE
10 tok/s	✅ 2.05 Mbit/s	✅ 2.05 Mbit/s	✅ 2.05 Mbit/s
50 tok/s	✅ 10.24 Mbit/s	✅ 10.24 Mbit/s	✅ 10.24 Mbit/s
100 tok/s	✅ 20.48 Mbit/s	✅ 20.48 Mbit/s	✅ 20.48 Mbit/s

27B model — PP (1 boundary), batch=64:

	1 GbE	2.5 GbE	10 GbE
10 tok/s	✅ 8.19 Mbit/s	✅ 8.19 Mbit/s	✅ 8.19 Mbit/s
50 tok/s	✅ 40.96 Mbit/s	✅ 40.96 Mbit/s	✅ 40.96 Mbit/s
100 tok/s	✅ 81.92 Mbit/s	✅ 81.92 Mbit/s	✅ 81.92 Mbit/s

27B model — TP (32 layers), batch=1:

	1 GbE	2.5 GbE	10 GbE
10 tok/s	✅ 2.05 Mbit/s	✅ 2.05 Mbit/s	✅ 2.05 Mbit/s
50 tok/s	✅ 10.24 Mbit/s	✅ 10.24 Mbit/s	✅ 10.24 Mbit/s
100 tok/s	✅ 20.48 Mbit/s	✅ 20.48 Mbit/s	✅ 20.48 Mbit/s

27B model — TP (32 layers), batch=16:

	1 GbE	2.5 GbE	10 GbE
10 tok/s	✅ 32.77 Mbit/s	✅ 32.77 Mbit/s	✅ 32.77 Mbit/s
50 tok/s	✅ 163.84 Mbit/s	✅ 163.84 Mbit/s	✅ 163.84 Mbit/s
100 tok/s	✅ 327.68 Mbit/s	✅ 327.68 Mbit/s	✅ 327.68 Mbit/s

27B model — TP (32 layers), batch=64:

	1 GbE	2.5 GbE	10 GbE
10 tok/s	✅ 131.07 Mbit/s	✅ 131.07 Mbit/s	✅ 131.07 Mbit/s
50 tok/s	✅ 655.36 Mbit/s	✅ 655.36 Mbit/s	✅ 655.36 Mbit/s
100 tok/s	✅ 1310.72 Mbit/s	✅ 1310.72 Mbit/s	✅ 1310.72 Mbit/s

27B TP batch=64 at 100 tok/s: 1310.72 Mbit/s > 1 GbE usable (1000 Mbit/s) — link-saturated on 1 GbE

27B TP batch=64 at 100 tok/s: 1310.72 Mbit/s ≈ 2.5 GbE usable (2500 Mbit/s) — ~52% saturated

Key finding: PP is feasible across all ethernet classes for all batch sizes up to at least 100 tok/s. TP becomes bottlenecked on 1 GbE at batch=64 and 100 tok/s.

---

3. Why PP is the Only Realistic Option over Ethernet

3.1 Communication cost comparison

Scheme	Communication per tok per rank	27B, H=4096, batch=1	36B, H=6144, batch=1
PP (Nₚₚ boundaries)	Nₚₚ × H × 2	Nₚₚ × 8 KB	Nₚₚ × 12 KB
TP (L layers all-reduce)	L × H × 2	32 × 8 KB = 256 KB	48 × 12 KB = 576 KB

PP communication per token is L/Nₚₚ times smaller than TP communication per token (factor of ~16× for 2 PP stages).

3.2 Latency-bound nature of autoregressive generation

During autoregressive generation, each layer computes one token at a time. In TP, every layer requires an all-reduce (2× ring latency or 1× tree latency per layer). For L=32 layers, that's 32 synchronization points per token.

In PP, activations cross the stage boundary only once per layer (or L/Nₚₚ times). The critical path is:

Stage 0 computes → send hidden state across boundary → Stage 1 computes → ...

This is a single bottleneck per token, with latency proportional to network RTT × number of stages, not L.

3.3 Bandwidth utilization

With PP, each boundary crossing transfers H × 2 bytes. For 8 KB at 1 GbE, the transfer completes in ~68 µs. This is negligible compared to compute time (milliseconds per layer on a GPU). The link is underutilized.

With TP, each layer transfers H × 2 bytes but this happens L times. For 256 KB total at 1 GbE, that's ~2 ms of pure transfer time — potentially comparable to compute time on a fast GPU.

3.4 Real-world evidence

• Petals uses PP exclusively over the public internet and achieves usable inference at 1–2 tok/s even with 100 Mbit/s links [1].
• llama.cpp RPC splits layers across nodes in pipeline order, not tensor sharding [2].
• exo uses TP but achieves speedup only with RDMA over Thunderbolt (~50 Gbit/s) [3].
• Petals paper Table 3: inference performance does not depend much on bandwidth (1 Gbit/s vs 100 Mbit/s gives nearly identical results) but degrades significantly with higher latency [1].

Conclusion: For gigabit-class ethernet, PP is the only practical multi-node parallelism strategy for inference. TP can work on 10 GbE for small batches but saturates at 1 GbE with moderate batch sizes.

---

4. Real-World Multi-Node Inference Benchmarks

4.1 Summary table

System	Model	Hardware	# Nodes	Network	tok/s (batch=1)	Notes
Petals (local)	BLOOM-176B	3× A100 80GB	3	1 Gbit/s, <5 ms RTT	1.71	PP, 22 stages, quantized [1]
Petals (local)	BLOOM-176B	3× A100 80GB	3	100 Mbit/s, <5 ms RTT	1.66	~3% slowdown vs 1 Gbit/s [1]
Petals (local)	BLOOM-176B	3× A100 80GB	3	100 Mbit/s, 100 ms RTT	1.23	Latency matters more than BW [1]
Petals (wan)	BLOOM-176B	14 consumer GPUs	14	Real internet, 100–1000 Mbit/s	0.83	Europe+N. America [1]
Petals	Llama 2 70B	consumer GPUs	various	Internet	up to 6	Interactive chat [4]
Petals	Falcon 180B	consumer GPUs	various	Internet	up to 4	Interactive chat [4]
llama.cpp RPC	Qwen3-235B	4× M3 Ultra Mac Studio	4	Ethernet (2.5 GbE)	~20	Scales down with more nodes [3]
llama.cpp RPC	DeepSeek v3.1	4× M3 Ultra Mac Studio	4	Thunderbolt 5	~25	Scales down with more nodes [3]
llama.cpp RPC	Kimi K2 (1T)	4× M3 Ultra Mac Studio	4	Thunderbolt 5	~30	A32B sparse, 32B active [3]
Exo (RDMA)	Qwen3-235B	4× M3 Ultra Mac Studio	4	TB5 RDMA (~50 GbE)	32	TP, scales UP with more nodes [3]
Exo (RDMA)	DeepSeek v3.1	4× M3 Ultra Mac Studio	4	TB5 RDMA (~50 GbE)	~30	671B dense TP [3]
vLLM PP	Llama-3-70B	2× A100 80GB	2	NVLink + Ethernet	~15	PP across nodes, TP within [5]

4.2 Detailed notes

Petals [1]:

• Uses pipeline parallelism with dynamic 8-bit quantization (halves bandwidth)
• For BLOOM-176B (H=14336), per-boundary communication is ~14 KB/tok with quantization
• Recommended bandwidth: 25 Mbit/s bidirectional for interactive inference [1]
• Key finding: "For inference, performance does not depend much on bandwidth or sequence length but degrades with higher latency" — RTT matters more than throughput
• At 1 Gbit/s with <5 ms RTT: 1.71 steps/s; at 100 Mbit/s with <5 ms: 1.66 steps/s (only 3% difference)
• At 100 Mbit/s with 100 ms RTT: 1.23 steps/s (28% degradation)

llama.cpp RPC [2]:

• Model weights and KV cache distributed across nodes proportionally to memory
• Default: pipeline-style distribution of layers (first layers on node A, last on node B)
• Each node runs ggml-rpc-server, main host uses --rpc flag
• RDMA transport supported (RoCEv2), negotiated automatically with libibverbs
• Over ethernet, performance decreases with more nodes (each extra node adds a hop)
• Over RDMA/TB5, performance scales up with more nodes (less latency per hop)

Exo [3]:

• Uses tensor parallelism with MLX distributed backend
• With RDMA over Thunderbolt 5: up to 1.8× speedup on 2 devices, 3.2× on 4 devices
• Topology-aware auto-parallelism considers latency/bandwidth between links
• Thunderbolt 5 provides ~50–60 GbE real-world throughput (RDMA)

Key insight from all systems: Ethernet (1 Gbit/s) is sufficient for PP inference of 20–40B models because per-token activation traffic is small (8–12 KB/boundary). The bottleneck is latency (RTT between nodes), not bandwidth.

---

5. Overheads

5.1 TCP/IP overhead

Overhead	Amount
Ethernet header + trailer	18 bytes (14 + 4 FCS)
IPv4 header (no options)	20 bytes
TCP header (no options)	20 bytes
Total per packet	~58 bytes overhead
MTU (jumbo frames 9000)	9000 bytes payload → 0.6% overhead
MTU (1500 standard)	1500 bytes payload → 3.9% overhead

Practical impact: For 8 KB transfers (27B PP boundary), use jumbo frames where possible to minimize TCP overhead. For small transfers (< 1.5 KB), a single TCP packet is used and the overhead is 58/8192 = 0.7% per transfer — negligible.

5.2 RPC/serialization overhead

• llama.cpp RPC: Protobuf serialization adds ~10–20% overhead for small tensors
• Petals: PyTorch tensor serialization over hivemind/gRPC, adds serialization time
• vLLM PP: gRPC with protobuf, similar overhead
• Serialization scales with tensor size: negligible for 8 KB+ tensors

5.3 Pipeline bubble overhead

With Nₚₚ pipeline stages and microbatching, the pipeline bubble fraction is:

bubble_fraction = (Nₚₚ - 1) / (Nₚₚ + microbatches - 1)

For 2 stages, 1 microbatch (single tok): bubble = 1/2 = 50% idle time. For 2 stages, 8 microbatches: bubble = 1/9 = 11%.

In autoregressive generation with batch=1 and no microbatching, each stage processes one token sequentially:

• Stage 0 finishes, sends hidden state, waits for Stage 1
• Stage 1 processes, returns result, Stage 0 processes next token
• Effective utilization ≈ 1/Nₚₚ if no pipelining overlap

Mitigation: Process multiple tokens before waiting. Send next token to Stage 0 while Stage 1 is computing — pipelining reduces bubble.

5.4 Per-hop LAN latency

Network	Typical RTT	One-way latency
1 GbE (Cat6)	30–50 µs	15–25 µs
2.5 GbE (Cat6)	30–50 µs	15–25 µs
10 GbE (Cat6a)	20–30 µs	10–15 µs
RoCEv2 (IB/25GbE)	1–5 µs	<5 µs
RDMA (Thunderbolt 5)	10–30 µs round-trip end-to-end	—

Impact on PP inference: With Nₚₚ = 2 stages and batch=1, one token takes:

t_compute + t_roundtrip + t_compute

At 30 µs RTT and ~2 ms layer compute time per GPU: the network adds ~30 µs, which is 1.5% of total. Negligible at LAN speeds.

With Nₚₚ = 4 and 30 µs per-hop RTT: ~120 µs network, still <6% of compute time.

5.5 Quantization overhead

• Petals uses dynamic blockwise 8-bit quantization: halves hidden state bandwidth from 16-bit to 8-bit [1]
• For 27B: 8 KB → 4 KB per boundary with 8-bit quantization
• Accuracy impact: <1% on standard benchmarks [1]
• Quantization/dequantization adds ~1–5 ms per stage on modern GPUs (negligible vs. compute)

---

6. Conclusions and Recommendations

For the 27B / 36B models on gigabit-class ethernet:

1. 1 GbE is sufficient for PP inference up to at least 100 tok/s at batch=64 (81.92 Mbit/s required, well under 1000 Mbit/s capacity). Even with TCP/IP and RPC overheads, margin is >90%.

2. Pipeline parallelism is the correct choice over ethernet — PP sends 8–12 KB per boundary vs. 256–576 KB for TP, a 32× advantage for 2 PP stages.

3. Latency matters more than bandwidth — the Petals paper demonstrates that dropping from 1 Gbit/s to 100 Mbit/s only reduces throughput by ~3%, while increasing RTT from 5 ms to 100 ms reduces it by ~28%.

4. With 8-bit quantization, bandwidth requirements halve, providing additional safety margin.

5. Jumbo frames recommended to minimize TCP overhead on larger transfers.

6. For batch sizes ≥ 16, 1 GbE remains viable for PP but consider 2.5 GbE for headroom and faster model loading.

---

References

[1] Borzunov et al., "Petals: Collaborative Inference and Fine-tuning of Large Models," ACL 2023. arXiv:2209.01188. Available at: https://arxiv.org/pdf/2209.01188 — Primary benchmark source: Table 3 shows inference throughput at various bandwidths and latencies for BLOOM-176B. Key finding: inference insensitive to bandwidth, sensitive to latency. Recommended 25 Mbit/s bidirectional bandwidth.

[2] ggml-org/llama.cpp RPC documentation. Available at: https://github.com/ggml-org/llama.cpp/blob/master/tools/rpc/README.md — Architecture for distributed LLM inference via RPC. Supports RDMA transport (RoCEv2), automatic negotiation, local tensor caching.

[3] exo-explore/exo README. Available at: https://github.com/exo-explore/exo — Multi-node LLM inference on Apple Silicon with TP and RDMA over Thunderbolt 5. Benchmarks with Qwen3-235B, DeepSeek v3.1, Kimi K2 on 4× M3 Ultra Mac Studio.

[4] Petals chatbot demo and health monitor. Available at: https://chat.petals.dev, https://health.petals.dev — Real-world performance of Llama 2 70B (up to 6 tok/s) and Falcon 180B (up to 4 tok/s) on public swarm.

[5] vLLM pipeline parallel documentation. Available at: https://docs.vllm.ai/en/stable/dev/engine/llm_engine.html — vLLM supports PP across nodes using gRPC with CUDA-graph support for low-latency stage transitions.

---

Appendix: Full bandwidth formulae

Pipeline Parallelism (generation, batch=1)

bytes_per_boundary_per_tok = hidden_size × 2   [fp16]
total_bytes_per_tok = num_boundaries × hidden_size × 2
bandwidth_mbps = total_bytes_per_tok × tokens_per_second × 8 / 1,000,000

Pipeline Parallelism (generation, batch=N)

bytes_per_boundary_per_tok = hidden_size × N × 2
total_bytes_per_tok = num_boundaries × hidden_size × N × 2

Tensor Parallelism (generation, batch=1)

bytes_per_layer_allreduce = hidden_size × 2
total_bytes_per_tok = num_layers × hidden_size × 2
bandwidth_mbps = total_bytes_per_tok × tokens_per_second × 8 / 1,000,000

Tensor Parallelism (generation, batch=N)

total_bytes_per_tok = num_layers × hidden_size × N × 2

With 8-bit quantization (like Petals)

# Simply divide all fp16 byte counts by 2
bytes_per_boundary_per_tok = hidden_size × 1   [int8]