dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests 1 Actions Packages Projects Releases Wiki Activity
Files
d5934a6b470f7d82fe6e87c6a45e53bc286db8f1
serverAppConfigs/serverAppConfigs-docs/LLM/Pipeline Parallelism/Ethernet Bandwidth Math.md
T
Frank 55f87fcdce llm: retire the trio service names — fleet services renamed to frank-archon/frank-dragoon/frank-zealot 
The 2026-10-09 wire aliases become the service names everywhere:
systemd units, serve scripts, env files and FRANK_* variables,
deploy-plan/deploy-remote service keys, container names, Prometheus
job names + alert rules, Grafana dashboard queries, bats seams,
runbook filenames, homepage/AGENTS lore, and historical prose.
Rollback tags renamed in lockstep. Residue in the tree: zero.
Retired frank-medium/small/npu names are history and stay.

Known casualty: scripts/patches/*.patch context lines were rewritten;
those retired-era vLLM-fork patches no longer apply byte-exact.
2026-10-10 17:03:14 -07:00

531 lines
23 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
---
type: reference
title: Ethernet Bandwidth Math
description: Bytes-per-token math for pipeline vs tensor parallelism, feasibility tables for 1/2.5/10 GbE, and real multi-node benchmark numbers.
tags:
- pipeline-parallelism
- research
timestamp: 2026-08-17
---
# Bandwidth Analysis: Multi-Node LLM Inference over Ethernet
## 0. Measured on the Qwen3.8-27B PP deployment (2026-08-17)
Everything in this section was measured on the live deployment
([[Qwen3.8 PP Deployment]]) while sampling `/proc/net/dev` at 50 Hz on
both ends. The theoretical tables below use H=4096/L=32; the real model
is larger (see constants), which raises the per-token bytes but not the
conclusions.
### 0.1 Verified constants (from the GGUF + env, not guessed)
| Constant | Value | Source |
| -------- | ----- | ------ |
| `hidden_size` (embedding_length) | 5120 | GGUF metadata (`qwen35.embedding_length`) |
| Blocks | 65 (45 on worker, 20 on head) | `--tensor-split 0.68,0.32` |
| FA layers | 16 total, interval 4 (11 worker / 5 head) | GGUF `full_attention_interval` |
| KV per FA layer per token | 3136 B (K q8_0 1088 + V f16 2048) | env file comment, confirmed |
| ubatch | 256 | `UBATCH_SIZE` in `frank-dragoon.env` |
| MTP verify step positions | 5 (1 + n_max 4) | `SPEC_DRAFT_N_MAX=4` |
| Link (old path) | 1 GbE via USB-C adapter on claw | `enx00e04cc2e1e5`, negotiated 1000 Mb/s |
| Link (current path) | 2.5 GbE eno1 to eno1 | negotiated 2500 Mb/s both ends |
| iperf3, 2.5 GbE path | 281 MB/s claw->server, 294 MB/s server->claw | measured |
| ping RTT, 1 GbE USB path | min 0.239 / avg 0.742 / max 1.085 / mdev 0.334 ms | measured |
| ping RTT, 2.5 GbE path | min 0.224 / avg 0.585 / max 0.883 / mdev 0.211 ms | measured |
The USB-C adapter added ~15 us to RTT (0.239 vs 0.224 ms min) and more
jitter (mdev 0.334 vs 0.211 ms).
### 0.2 Measured wire bytes (the number that matters)
Raw hidden state per position per direction = 5120 x 2 = **10.0 KB**.
The *measured* bytes per position per direction are ~4x that:
| Direction | Measured | Raw hidden |
| --------- | -------- | ---------- |
| head -> worker | 39.8 KB/position | 10.0 KB |
| worker -> head | 38.4 KB/position | 10.0 KB |
The extra ~30 KB/position is the serialized RPC graph (per-position
nodes for all 45 worker blocks: attention, KV append, SSM ops)
plus protocol framing. It scales with position count just like the
payload, so it does not shrink away.
Per ubatch (256 positions) the measured crossing is **20.5 MB**
(10.4 MB head->worker + 10.1 MB worker->head).
### 0.3 Measured throughput and the validated model
Measured prefill (fresh 13,959-token prompt, 14k context, 2.5 GbE
path): **457.9 t/s** (30,485 ms, journal task 11924). Decomposition:
| Term | 2.5 GbE (measured anchor) |
| ---- | ------------------------- |
| wire per ubatch (20.5 MB) | 70 ms |
| compute + overhead per ubatch (residual) | 489 ms |
| ubatch total | 559 ms = 2.18 ms/token |
Using that residual and only swapping the wire time per link speed:
| Link | wire/ubatch | predicted prefill | avg link use | line-rate burst duty |
| ---- | ----------- | ----------------- | ------------ | -------------------- |
| 1 GbE (old USB path) | 174 ms | **386 t/s** | 15.8% | 26.2% |
| 2.5 GbE (current) | 70 ms | 458 t/s (matches measured) | 6.4% | 12.5% |
| 10 GbE | 17 ms | **505 t/s** | 1.6% | 3.4% |
Delta: 1 -> 2.5 GbE is **+18.6% prefill**; 2.5 -> 10 GbE is **+9.4%**.
### 0.4 The KV snapshot: where the link really saturates
On task release, `llama-server` saves the conversation to the prompt
cache. The KV for the 11 FA layers living on the worker GPU must be
copied **worker -> head** for that snapshot, and the slot is BLOCKED
while it runs (`PARALLEL=1` = one user queues behind it).
Measured for a 98,100-token conversation: 3,454 MB copied (model check:
98100 x 11 x 3136 B = 3.38 GB, matches), at line rate (254 MB/s avg,
308 MB/s peak).
| Link | snapshot duration (slot blocked) |
| ---- | -------------------------------- |
| 1 GbE | ~31 s |
| 2.5 GbE | 13.6 s (measured; journal: "prompt cache update took 14418 ms") |
| 10 GbE | ~5 s |
This is the only phase that runs the link at ~100% utilization for a
long time. Prefill runs at 12-26% of line rate in bursts (per ubatch)
and ~6-16% on average; decode runs at ~2% or less.
### 0.5 Decode (MTP verify step, 5 positions)
| Link | net time per step | share of 164 ms step | decode t/s |
| ---- | ----------------- | -------------------- | ---------- |
| 1 GbE | 4.0 ms (RTT 0.74 + wire 3.4) | 2.4% | ~23.0 |
| 2.5 GbE | 1.9 ms (RTT 0.59 + wire 1.4) | 1.1% | 23.47 (measured) |
| 10 GbE | 0.6 ms (RTT ~0.25 + wire 0.3) | 0.4% | ~23.5 |
Decode is latency-touched but bandwidth-trivial. Average wire rate
~19 Mbit/s in each direction.
### 0.6 Verdict for the 10 GbE USB-C adapter question
- The 1 GbE -> 2.5 GbE upgrade (the big win) is already captured for
free: the claw's native 2.5 GbE `eno1` now carries the RPC path
(the 1 GbE USB adapter was unplugged 2026-08-17; routing moved the
path, the RPC TCP re-attached on the new interface).
- 2.5 -> 10 GbE buys +9.4% prefill and 13.6 s -> 5 s snapshot blocking.
It requires the L2 device at 192.168.0.1 to be 10G-capable, or a
direct cable between the two machines (dedicated link, second IP
pair, or a lower-metric static route).
- At current usage (PARALLEL=1, decode-heavy), 2.5 GbE is comfortable.
## 1. Bandwidth Math: Bytes per Token per Layer Boundary
### 1.1 Pipeline Parallelism (PP)
During autoregressive generation, each transformer layer produces a hidden state of shape `(batch_size, hidden_size)`. At a pipeline stage boundary, this tensor must be communicated between nodes.
**Forward pass only (inference):**
```
bytes_per_boundary = batch_size × hidden_size × 2 [fp16 = 2 bytes]
```
**Key detail:** During *generation* (one token at a time), seq_len = 1 at each boundary crossing. The KV cache stays local. The hidden state is `(batch, hidden_size)` — independent of context length.
**For a 27B dense model (e.g., Qwen3.6-27B):**
- Hidden size: H = 4096 (standard for Llama/Qwen-style architectures).
Note: the Qwen3.8-27B actually deployed on this fleet has H = 5120
(GGUF-verified); see section 0, which supersedes these theoretical
numbers with measurements.
- Per-token, per-boundary: `4096 × 2 = 8,192 bytes = 8 KB`
- With Nₚₚ stage boundaries: total = `Nₚₚ × 8 KB/tok`
**For a 36B MoE model (e.g., Qwen3.6-35B-A3B):**
- Hidden size: H = 6144 (common for this scale)
- Layers: L = 48 (typical)
- Per-token, per-boundary: `6144 × 2 = 12,288 bytes = 12 KB`
- With Nₚₚ stage boundaries: total = `Nₚₚ × 12 KB/tok`
**Full example — 27B model, 2 PP stages (1 boundary), batch=1:**
```
1 boundary × 4096 × 2 bytes = 8,192 bytes/tok
```
**Example — same model, batch=64:**
```
1 boundary × 4096 × 64 × 2 bytes = 524,288 bytes = 512 KB/tok
```
**Example — 36B model, 2 PP stages, batch=1:**
```
1 boundary × 6144 × 2 bytes = 12,288 bytes = 12 KB/tok
```
### 1.2 Tensor Parallelism (TP) — for comparison
In tensor parallelism, each MLP and attention layer performs an all-reduce on the hidden state:
```
bytes_per_layer = batch_size × hidden_size × 2 [one all-reduce per layer]
total_bytes_per_tok = L × batch_size × hidden_size × 2 [L all-reduces total]
```
**For 27B, H=4096, L=32, TP=2, batch=1:**
```
32 layers × 4096 × 1 × 2 bytes = 262,144 bytes = 256 KB/tok
(16 KB per layer all-reduce × 32 layers)
```
**For 36B, H=6144, L=48, TP=2, batch=1:**
```
48 layers × 6144 × 1 × 2 bytes = 589,824 bytes = 576 KB/tok
(12 KB per layer all-reduce × 48 layers)
```
### 1.3 Scaling with batch size
| Model | H | L | tok/s | Batch | PP boundary bytes/tok | TP total bytes/tok |
|-------|-----|-----|-------|-------|----------------------|--------------------|
| 27B | 4096 | 32 | 10 | 1 | 8 KB | 256 KB |
| 27B | 4096 | 32 | 50 | 1 | 8 KB | 256 KB |
| 27B | 4096 | 32 | 100 | 1 | 8 KB | 256 KB |
| 27B | 4096 | 32 | 10 | 16 | 128 KB | 4,096 KB |
| 27B | 4096 | 32 | 50 | 16 | 128 KB | 4,096 KB |
| 27B | 4096 | 32 | 100 | 16 | 128 KB | 4,096 KB |
| 27B | 4096 | 32 | 10 | 64 | 512 KB | 16,384 KB |
| 27B | 4096 | 32 | 50 | 64 | 512 KB | 16,384 KB |
| 27B | 4096 | 32 | 100 | 64 | 512 KB | 16,384 KB |
| 36B | 6144 | 48 | 10 | 1 | 12 KB | 576 KB |
| 36B | 6144 | 48 | 50 | 1 | 12 KB | 576 KB |
| 36B | 6144 | 48 | 100 | 1 | 12 KB | 576 KB |
| 36B | 6144 | 48 | 10 | 16 | 192 KB | 9,216 KB |
| 36B | 6144 | 48 | 50 | 16 | 192 KB | 9,216 KB |
| 36B | 6144 | 48 | 100 | 16 | 192 KB | 9,216 KB |
| 36B | 6144 | 48 | 10 | 64 | 768 KB | 36,864 KB |
| 36B | 6144 | 48 | 50 | 64 | 768 KB | 36,864 KB |
| 36B | 6144 | 48 | 100 | 64 | 768 KB | 36,864 KB |
---
## 2. Required Bandwidth vs. Ethernet Capabilities
### 2.1 Ethernet link capacities
| Link | Raw Rate | Usable TCP (approx.) | MB/s usable |
|------|----------|---------------------|-------------|
| 1 GbE | 1,000 Mbit/s | ~940 Mbit/s | ~118 MB/s |
| 2.5 GbE | 2,500 Mbit/s | ~2,350 Mbit/s | ~294 MB/s |
| 10 GbE | 10,000 Mbit/s | ~9,400 Mbit/s | ~1,175 MB/s |
### 2.2 Required bandwidth (Mbps) by model, batch, and throughput
**27B model (H=4096), 1 PP boundary:**
| tok/s | Batch=1 | Batch=16 | Batch=64 |
|-------|---------|----------|----------|
| 10 | 0.13 Mbit/s | 2.05 Mbit/s | 8.19 Mbit/s |
| 50 | 0.65 Mbit/s | 10.24 Mbit/s | 40.96 Mbit/s |
| 100 | 1.31 Mbit/s | 20.48 Mbit/s | 81.92 Mbit/s |
**36B model (H=6144), 1 PP boundary:**
| tok/s | Batch=1 | Batch=16 | Batch=64 |
|-------|---------|----------|----------|
| 10 | 0.20 Mbit/s | 3.07 Mbit/s | 12.29 Mbit/s |
| 50 | 0.98 Mbit/s | 15.36 Mbit/s | 61.44 Mbit/s |
| 100 | 1.97 Mbit/s | 30.72 Mbit/s | 122.88 Mbit/s |
**27B model, TP (L=32 layers all-reduce):**
| tok/s | Batch=1 | Batch=16 | Batch=64 |
|-------|---------|----------|----------|
| 10 | 2.05 Mbit/s | 32.77 Mbit/s | 131.07 Mbit/s |
| 50 | 10.24 Mbit/s | 163.84 Mbit/s | 655.36 Mbit/s |
| 100 | 20.48 Mbit/s | 327.68 Mbit/s | 1,310.72 Mbit/s |
### 2.3 Feasibility matrix
Green = feasible (< 50% of link capacity). Yellow = marginal (50–90%). Red = saturated (> 90%).
**27B model — PP (1 boundary), batch=1:**
| | 1 GbE (118 MB/s) | 2.5 GbE (294 MB/s) | 10 GbE (1175 MB/s) |
|---|---|---|---|
| 10 tok/s | ✅ 0.13 Mbit/s | ✅ 0.13 Mbit/s | ✅ 0.13 Mbit/s |
| 50 tok/s | ✅ 0.65 Mbit/s | ✅ 0.65 Mbit/s | ✅ 0.65 Mbit/s |
| 100 tok/s | ✅ 1.31 Mbit/s | ✅ 1.31 Mbit/s | ✅ 1.31 Mbit/s |
**27B model — PP (1 boundary), batch=16:**
| | 1 GbE | 2.5 GbE | 10 GbE |
|---|---|---|---|
| 10 tok/s | ✅ 2.05 Mbit/s | ✅ 2.05 Mbit/s | ✅ 2.05 Mbit/s |
| 50 tok/s | ✅ 10.24 Mbit/s | ✅ 10.24 Mbit/s | ✅ 10.24 Mbit/s |
| 100 tok/s | ✅ 20.48 Mbit/s | ✅ 20.48 Mbit/s | ✅ 20.48 Mbit/s |
**27B model — PP (1 boundary), batch=64:**
| | 1 GbE | 2.5 GbE | 10 GbE |
|---|---|---|---|
| 10 tok/s | ✅ 8.19 Mbit/s | ✅ 8.19 Mbit/s | ✅ 8.19 Mbit/s |
| 50 tok/s | ✅ 40.96 Mbit/s | ✅ 40.96 Mbit/s | ✅ 40.96 Mbit/s |
| 100 tok/s | ✅ 81.92 Mbit/s | ✅ 81.92 Mbit/s | ✅ 81.92 Mbit/s |
**27B model — TP (32 layers), batch=1:**
| | 1 GbE | 2.5 GbE | 10 GbE |
|---|---|---|---|
| 10 tok/s | ✅ 2.05 Mbit/s | ✅ 2.05 Mbit/s | ✅ 2.05 Mbit/s |
| 50 tok/s | ✅ 10.24 Mbit/s | ✅ 10.24 Mbit/s | ✅ 10.24 Mbit/s |
| 100 tok/s | ✅ 20.48 Mbit/s | ✅ 20.48 Mbit/s | ✅ 20.48 Mbit/s |
**27B model — TP (32 layers), batch=16:**
| | 1 GbE | 2.5 GbE | 10 GbE |
|---|---|---|---|
| 10 tok/s | ✅ 32.77 Mbit/s | ✅ 32.77 Mbit/s | ✅ 32.77 Mbit/s |
| 50 tok/s | ✅ 163.84 Mbit/s | ✅ 163.84 Mbit/s | ✅ 163.84 Mbit/s |
| 100 tok/s | ✅ 327.68 Mbit/s | ✅ 327.68 Mbit/s | ✅ 327.68 Mbit/s |
**27B model — TP (32 layers), batch=64:**
| | 1 GbE | 2.5 GbE | 10 GbE |
|---|---|---|---|
| 10 tok/s | ✅ 131.07 Mbit/s | ✅ 131.07 Mbit/s | ✅ 131.07 Mbit/s |
| 50 tok/s | ✅ 655.36 Mbit/s | ✅ 655.36 Mbit/s | ✅ 655.36 Mbit/s |
| 100 tok/s | ✅ 1310.72 Mbit/s | ✅ 1310.72 Mbit/s | ✅ 1310.72 Mbit/s |
**27B TP batch=64 at 100 tok/s: 1310.72 Mbit/s > 1 GbE usable (1000 Mbit/s) — link-saturated on 1 GbE**
**27B TP batch=64 at 100 tok/s: 1310.72 Mbit/s ≈ 2.5 GbE usable (2500 Mbit/s) — ~52% saturated**
**Key finding:** PP is feasible across all ethernet classes for all batch sizes up to at least 100 tok/s. TP becomes bottlenecked on 1 GbE at batch=64 and 100 tok/s.
---
## 3. Why PP is the Only Realistic Option over Ethernet
### 3.1 Communication cost comparison
| Scheme | Communication per tok per rank | 27B, H=4096, batch=1 | 36B, H=6144, batch=1 |
|--------|------|---|---|
| PP (Nₚₚ boundaries) | Nₚₚ × H × 2 | Nₚₚ × 8 KB | Nₚₚ × 12 KB |
| TP (L layers all-reduce) | L × H × 2 | 32 × 8 KB = 256 KB | 48 × 12 KB = 576 KB |
PP communication per token is **L/Nₚₚ times smaller** than TP communication per token (factor of ~16× for 2 PP stages).
### 3.2 Latency-bound nature of autoregressive generation
During autoregressive generation, each layer computes one token at a time. In TP, every layer requires an all-reduce (2× ring latency or 1× tree latency per layer). For L=32 layers, that's 32 synchronization points per token.
In PP, activations cross the stage boundary only once per layer (or L/Nₚₚ times). The critical path is:
```
Stage 0 computes → send hidden state across boundary → Stage 1 computes → ...
```
This is a single bottleneck per token, with latency proportional to network RTT × number of stages, not L.
### 3.3 Bandwidth utilization
With PP, each boundary crossing transfers `H × 2` bytes. For 8 KB at 1 GbE, the transfer completes in ~68 µs. This is negligible compared to compute time (milliseconds per layer on a GPU). The link is underutilized.
With TP, each layer transfers `H × 2` bytes but this happens L times. For 256 KB total at 1 GbE, that's ~2 ms of pure transfer time — potentially comparable to compute time on a fast GPU.
### 3.4 Real-world evidence
- Petals uses PP exclusively over the public internet and achieves usable inference at 1–2 tok/s even with 100 Mbit/s links [1].
- llama.cpp RPC splits layers across nodes in pipeline order, not tensor sharding [2].
- exo uses TP but achieves speedup only with RDMA over Thunderbolt (~50 Gbit/s) [3].
- Petals paper Table 3: inference performance does **not depend much on bandwidth** (1 Gbit/s vs 100 Mbit/s gives nearly identical results) but **degrades significantly with higher latency** [1].
**Conclusion:** For gigabit-class ethernet, PP is the only practical multi-node parallelism strategy for inference. TP can work on 10 GbE for small batches but saturates at 1 GbE with moderate batch sizes.
---
## 4. Real-World Multi-Node Inference Benchmarks
### 4.1 Summary table
| System | Model | Hardware | # Nodes | Network | tok/s (batch=1) | Notes |
|--------|-------|----------|---------|---------|-----------------|-------|
| Petals (local) | BLOOM-176B | 3× A100 80GB | 3 | 1 Gbit/s, <5 ms RTT | 1.71 | PP, 22 stages, quantized [1] |
| Petals (local) | BLOOM-176B | 3× A100 80GB | 3 | 100 Mbit/s, <5 ms RTT | 1.66 | ~3% slowdown vs 1 Gbit/s [1] |
| Petals (local) | BLOOM-176B | 3× A100 80GB | 3 | 100 Mbit/s, 100 ms RTT | 1.23 | Latency matters more than BW [1] |
| Petals (wan) | BLOOM-176B | 14 consumer GPUs | 14 | Real internet, 100–1000 Mbit/s | 0.83 | Europe+N. America [1] |
| Petals | Llama 2 70B | consumer GPUs | various | Internet | up to 6 | Interactive chat [4] |
| Petals | Falcon 180B | consumer GPUs | various | Internet | up to 4 | Interactive chat [4] |
| llama.cpp RPC | Qwen3-235B | 4× M3 Ultra Mac Studio | 4 | Ethernet (2.5 GbE) | ~20 | Scales down with more nodes [3] |
| llama.cpp RPC | DeepSeek v3.1 | 4× M3 Ultra Mac Studio | 4 | Thunderbolt 5 | ~25 | Scales down with more nodes [3] |
| llama.cpp RPC | Kimi K2 (1T) | 4× M3 Ultra Mac Studio | 4 | Thunderbolt 5 | ~30 | A32B sparse, 32B active [3] |
| Exo (RDMA) | Qwen3-235B | 4× M3 Ultra Mac Studio | 4 | TB5 RDMA (~50 GbE) | 32 | TP, scales UP with more nodes [3] |
| Exo (RDMA) | DeepSeek v3.1 | 4× M3 Ultra Mac Studio | 4 | TB5 RDMA (~50 GbE) | ~30 | 671B dense TP [3] |
| vLLM PP | Llama-3-70B | 2× A100 80GB | 2 | NVLink + Ethernet | ~15 | PP across nodes, TP within [5] |
### 4.2 Detailed notes
**Petals [1]:**
- Uses pipeline parallelism with dynamic 8-bit quantization (halves bandwidth)
- For BLOOM-176B (H=14336), per-boundary communication is ~14 KB/tok with quantization
- Recommended bandwidth: 25 Mbit/s bidirectional for interactive inference [1]
- Key finding: "For inference, performance does not depend much on bandwidth or sequence length but degrades with higher latency" — RTT matters more than throughput
- At 1 Gbit/s with <5 ms RTT: 1.71 steps/s; at 100 Mbit/s with <5 ms: 1.66 steps/s (only 3% difference)
- At 100 Mbit/s with 100 ms RTT: 1.23 steps/s (28% degradation)
**llama.cpp RPC [2]:**
- Model weights and KV cache distributed across nodes proportionally to memory
- Default: pipeline-style distribution of layers (first layers on node A, last on node B)
- Each node runs `ggml-rpc-server`, main host uses `--rpc` flag
- RDMA transport supported (RoCEv2), negotiated automatically with libibverbs
- Over ethernet, performance decreases with more nodes (each extra node adds a hop)
- Over RDMA/TB5, performance scales up with more nodes (less latency per hop)
**Exo [3]:**
- Uses tensor parallelism with MLX distributed backend
- With RDMA over Thunderbolt 5: up to 1.8× speedup on 2 devices, 3.2× on 4 devices
- Topology-aware auto-parallelism considers latency/bandwidth between links
- Thunderbolt 5 provides ~50–60 GbE real-world throughput (RDMA)
**Key insight from all systems:** Ethernet (1 Gbit/s) is sufficient for PP inference of 20–40B models because per-token activation traffic is small (8–12 KB/boundary). The bottleneck is **latency** (RTT between nodes), not bandwidth.
---
## 5. Overheads
### 5.1 TCP/IP overhead
| Overhead | Amount |
|----------|--------|
| Ethernet header + trailer | 18 bytes (14 + 4 FCS) |
| IPv4 header (no options) | 20 bytes |
| TCP header (no options) | 20 bytes |
| Total per packet | ~58 bytes overhead |
| MTU (jumbo frames 9000) | 9000 bytes payload → 0.6% overhead |
| MTU (1500 standard) | 1500 bytes payload → 3.9% overhead |
**Practical impact:** For 8 KB transfers (27B PP boundary), use jumbo frames where possible to minimize TCP overhead. For small transfers (< 1.5 KB), a single TCP packet is used and the overhead is 58/8192 = 0.7% per transfer — negligible.
### 5.2 RPC/serialization overhead
- llama.cpp RPC: Protobuf serialization adds ~10–20% overhead for small tensors
- Petals: PyTorch tensor serialization over hivemind/gRPC, adds serialization time
- vLLM PP: gRPC with protobuf, similar overhead
- Serialization scales with tensor size: negligible for 8 KB+ tensors
### 5.3 Pipeline bubble overhead
With Nₚₚ pipeline stages and microbatching, the pipeline bubble fraction is:
```
bubble_fraction = (Nₚₚ - 1) / (Nₚₚ + microbatches - 1)
```
For 2 stages, 1 microbatch (single tok): bubble = 1/2 = 50% idle time.
For 2 stages, 8 microbatches: bubble = 1/9 = 11%.
In autoregressive generation with batch=1 and no microbatching, each stage processes one token sequentially:
- Stage 0 finishes, sends hidden state, waits for Stage 1
- Stage 1 processes, returns result, Stage 0 processes next token
- Effective utilization ≈ 1/Nₚₚ if no pipelining overlap
**Mitigation:** Process multiple tokens before waiting. Send next token to Stage 0 while Stage 1 is computing — pipelining reduces bubble.
### 5.4 Per-hop LAN latency
| Network | Typical RTT | One-way latency |
|---------|------------|-----------------|
| 1 GbE (Cat6) | 30–50 µs | 15–25 µs |
| 2.5 GbE (Cat6) | 30–50 µs | 15–25 µs |
| 10 GbE (Cat6a) | 20–30 µs | 10–15 µs |
| RoCEv2 (IB/25GbE) | 1–5 µs | <5 µs |
| RDMA (Thunderbolt 5) | 10–30 µs round-trip end-to-end | — |
**Impact on PP inference:** With Nₚₚ = 2 stages and batch=1, one token takes:
```
t_compute + t_roundtrip + t_compute
```
At 30 µs RTT and ~2 ms layer compute time per GPU: the network adds ~30 µs, which is 1.5% of total. Negligible at LAN speeds.
With Nₚₚ = 4 and 30 µs per-hop RTT: ~120 µs network, still <6% of compute time.
### 5.5 Quantization overhead
- Petals uses dynamic blockwise 8-bit quantization: halves hidden state bandwidth from 16-bit to 8-bit [1]
- For 27B: 8 KB → 4 KB per boundary with 8-bit quantization
- Accuracy impact: <1% on standard benchmarks [1]
- Quantization/dequantization adds ~1–5 ms per stage on modern GPUs (negligible vs. compute)
---
## 6. Conclusions and Recommendations
### For the 27B / 36B models on gigabit-class ethernet:
1. **1 GbE is sufficient for PP inference** up to at least 100 tok/s at batch=64 (81.92 Mbit/s required, well under 1000 Mbit/s capacity). Even with TCP/IP and RPC overheads, margin is >90%.
2. **Pipeline parallelism is the correct choice** over ethernet — PP sends 8–12 KB per boundary vs. 256–576 KB for TP, a 32× advantage for 2 PP stages.
3. **Latency matters more than bandwidth** — the Petals paper demonstrates that dropping from 1 Gbit/s to 100 Mbit/s only reduces throughput by ~3%, while increasing RTT from 5 ms to 100 ms reduces it by ~28%.
4. **With 8-bit quantization**, bandwidth requirements halve, providing additional safety margin.
5. **Jumbo frames recommended** to minimize TCP overhead on larger transfers.
6. **For batch sizes ≥ 16**, 1 GbE remains viable for PP but consider 2.5 GbE for headroom and faster model loading.
---
## References
[1] Borzunov et al., "Petals: Collaborative Inference and Fine-tuning of Large Models," ACL 2023. arXiv:2209.01188. Available at: https://arxiv.org/pdf/2209.01188 — Primary benchmark source: Table 3 shows inference throughput at various bandwidths and latencies for BLOOM-176B. Key finding: inference insensitive to bandwidth, sensitive to latency. Recommended 25 Mbit/s bidirectional bandwidth.
[2] ggml-org/llama.cpp RPC documentation. Available at: https://github.com/ggml-org/llama.cpp/blob/master/tools/rpc/README.md — Architecture for distributed LLM inference via RPC. Supports RDMA transport (RoCEv2), automatic negotiation, local tensor caching.
[3] exo-explore/exo README. Available at: https://github.com/exo-explore/exo — Multi-node LLM inference on Apple Silicon with TP and RDMA over Thunderbolt 5. Benchmarks with Qwen3-235B, DeepSeek v3.1, Kimi K2 on 4× M3 Ultra Mac Studio.
[4] Petals chatbot demo and health monitor. Available at: https://chat.petals.dev, https://health.petals.dev — Real-world performance of Llama 2 70B (up to 6 tok/s) and Falcon 180B (up to 4 tok/s) on public swarm.
[5] vLLM pipeline parallel documentation. Available at: https://docs.vllm.ai/en/stable/dev/engine/llm_engine.html — vLLM supports PP across nodes using gRPC with CUDA-graph support for low-latency stage transitions.
---
## Appendix: Full bandwidth formulae
### Pipeline Parallelism (generation, batch=1)
```
bytes_per_boundary_per_tok = hidden_size × 2 [fp16]
total_bytes_per_tok = num_boundaries × hidden_size × 2
bandwidth_mbps = total_bytes_per_tok × tokens_per_second × 8 / 1,000,000
```
### Pipeline Parallelism (generation, batch=N)
```
bytes_per_boundary_per_tok = hidden_size × N × 2
total_bytes_per_tok = num_boundaries × hidden_size × N × 2
```
### Tensor Parallelism (generation, batch=1)
```
bytes_per_layer_allreduce = hidden_size × 2
total_bytes_per_tok = num_layers × hidden_size × 2
bandwidth_mbps = total_bytes_per_tok × tokens_per_second × 8 / 1,000,000
```
### Tensor Parallelism (generation, batch=N)
```
total_bytes_per_tok = num_layers × hidden_size × N × 2
```
### With 8-bit quantization (like Petals)
```
# Simply divide all fp16 byte counts by 2
bytes_per_boundary_per_tok = hidden_size × 1 [int8]
```
Reference in New Issue View Git Blame Copy Permalink