docs: pipeline parallelism research + educational sample script set
Compiles the 5-subagent research fan-out (llama.cpp RPC, vLLM PP, bandwidth math, mixed-vendor experiments, Qwen3.6-27B spec methods) into the KB under docs/Pipeline Parallelism/ with an index, expected performance estimates (1/2.5/10 GbE), and an EDUCATIONAL-ONLY two-node Qwen3.6-27B Q8 split script set (MTP/DFlash/ngram-mod variants, DSpark noted as vLLM-only). Nothing here is deployed or intended to run.
This commit is contained in:
Frank committed
1 parent
8d1d9baa82
commit
db69fc8192
9 files changed
+1915
No files matched your search
@@ -0,0 +1,260 @@
|
||||
---
|
||||
type: reference
|
||||
title: vLLM Pipeline Parallelism
|
||||
description: vLLM multi-node pipeline parallelism: flags, Ray/mp backends, interconnect requirements, PP bubble, and PP + speculative decoding support.
|
||||
tags:
|
||||
- pipeline-parallelism
|
||||
- research
|
||||
timestamp: 2026-08-14
|
||||
---
|
||||
|
||||
# Research: vLLM Pipeline Parallelism Across Multiple Machines over Ethernet
|
||||
|
||||
## Summary
|
||||
|
||||
vLLM supports multi-node pipeline parallelism (PP) via `--pipeline-parallel-size` combined with `--tensor-parallel-size`, using either Ray (`--distributed-executor-backend ray`) or native multiprocessing (`--distributed-executor-backend mp` with `--nnodes`, `--node-rank`, `--master-addr`). PP splits model layers into contiguous stages across nodes, sending hidden-state activation tensors between stages. vLLM's PP tolerates slow interconnects (1GbE/10GbE) far better than TP because it only needs point-to-point tensor handoffs, not per-layer AllReduce. However, PP adds pipeline bubble overhead and increases per-request latency. As of vLLM <=0.15.0, PP and speculative decoding are **not composable**.
|
||||
|
||||
## Findings
|
||||
|
||||
### 1. vLLM Multi-Node Flags
|
||||
|
||||
**CLI (`vllm serve`):**
|
||||
|
||||
```bash
|
||||
# Ray backend (multi-node):
|
||||
vllm serve /path/to/model \
|
||||
--tensor-parallel-size 1 \
|
||||
--pipeline-parallel-size 2 \
|
||||
--distributed-executor-backend ray
|
||||
|
||||
# Multiprocessing backend (multi-node):
|
||||
# On node 0 (head):
|
||||
vllm serve /path/to/model \
|
||||
--tensor-parallel-size 1 \
|
||||
--pipeline-parallel-size 2 \
|
||||
--nnodes 2 \
|
||||
--node-rank 0 \
|
||||
--master-addr <NODE0_IP>
|
||||
|
||||
# On node 1:
|
||||
vllm serve /path/to/model \
|
||||
--tensor-parallel-size 1 \
|
||||
--pipeline-parallel-size 2 \
|
||||
--nnodes 2 \
|
||||
--node-rank 1 \
|
||||
--master-addr <NODE0_IP> \
|
||||
--headless
|
||||
```
|
||||
|
||||
**Python API (`LLM` class):**
|
||||
|
||||
```python
|
||||
from vllm import LLM
|
||||
|
||||
llm = LLM(
|
||||
model="/path/to/model",
|
||||
tensor_parallel_size=1,
|
||||
pipeline_parallel_size=2,
|
||||
# For multi-node, set these via EngineArgs kwargs:
|
||||
distributed_executor_backend="ray", # or "mp"
|
||||
# For mp backend:
|
||||
# nnodes=2, node_rank=0, master_addr="<ip>"
|
||||
)
|
||||
```
|
||||
|
||||
**2-node setup examples:**
|
||||
|
||||
| Configuration | tp_size | pp_size | Description |
|
||||
|---|---|---|---|
|
||||
| Dense 27B on 2x single-GPU nodes | 1 | 2 | Layers split evenly across 2 nodes |
|
||||
| 27B on 2x 2-GPU nodes | 1 | 2 | Each node runs 1 PP stage |
|
||||
| 70B on 2x 2-GPU nodes | 2 | 2 | Each PP stage has 2 GPUs in TP |
|
||||
|
||||
**EngineArgs key fields** (from `vllm/engine/arg_utils.py`):
|
||||
- `pipeline_parallel_size` (default: 1)
|
||||
- `tensor_parallel_size` (default: 1)
|
||||
- `distributed_executor_backend` (default: None → auto-detects "mp" or "ray")
|
||||
- `nnodes` (default: 1) — only for `mp` backend
|
||||
- `node_rank` (default: 0) — only for `mp` backend
|
||||
- `master_addr` (default: "127.0.0.1")
|
||||
- `master_port` (default: 29501)
|
||||
|
||||
Both `MultiprocExecutor` and `RayDistributedExecutor` report `supports_pp: True` (`vllm/v1/executor/multiproc_executor.py` line ~88, `vllm/v1/executor/ray_executor.py` line ~67).
|
||||
|
||||
**Ray cluster setup** (`docs/examples/ray_serving/multi-node-serving.sh`):
|
||||
```bash
|
||||
# Head node:
|
||||
ray start --head --port=6379
|
||||
# Worker node:
|
||||
ray start --address='<HEAD_IP>:6379'
|
||||
```
|
||||
|
||||
**Source:** `docs/serving/parallelism_scaling.md`, `vllm/config/parallel.py`, `vllm/engine/arg_utils.py`
|
||||
|
||||
### 2. Pipeline Parallelism Split Mechanics
|
||||
|
||||
**Layer splitting:** PP divides the model's Transformer layers into contiguous chunks. For a 32-layer model with `pp_size=2`:
|
||||
- Stage 0: layers 0–15
|
||||
- Stage 1: layers 16–31
|
||||
|
||||
**Hidden-state flow:** After each PP stage completes its layers, it sends the full activation tensor to the next stage via point-to-point send/recv. Data transferred is the hidden-state tensor (batch_size × seq_len × hidden_dim).
|
||||
|
||||
**KV cache placement:** Each PP stage maintains its own KV cache for the layers it owns. With PP=2 on a 32-layer model, each stage holds half the layers' KV cache. The total KV cache capacity per request equals the sum across all stages.
|
||||
|
||||
**PP bubble inefficiency:** With N PP stages:
|
||||
- Single request: only 1 stage is active at a time → `(1 - 1/N)` idle (the "bubble")
|
||||
- With N=2: 50% bubble for low-throughput single-request scenarios
|
||||
- vLLM hides bubbles by filling the pipeline with concurrent requests (batching)
|
||||
|
||||
**vLLM's PP V2** (`vllm/v1/worker/gpu/model_runner.py`) introduces a modular `PPHandler` class. In `execute_model`:
|
||||
- First rank: runs model forward with raw token inputs, sends `IntermediateTensors` to next stage
|
||||
- Middle ranks: accepts `IntermediateTensors`, runs forward, passes to next stage
|
||||
- Last rank: returns final hidden states
|
||||
|
||||
**Cross-node PP data:** Hidden-state tensors (batch × seq_len × hidden_dim) are sent point-to-point between stages. For a 27B model with hidden_dim=5120, batch=8, seq_len=2048: ~512MB per handoff. This happens once per forward pass per stage boundary.
|
||||
|
||||
**Source:** `docs/serving/parallelism_scaling.md`, `docs/features/pipeline_parallel.md`, `vllm/v1/executor/multiproc_executor.py`, PR #33960
|
||||
|
||||
### 3. Interconnect Requirements
|
||||
|
||||
**Official vLLM guidance** (from `docs/serving/parallelism_scaling.md` and the blog):
|
||||
|
||||
- **TP within a node:** Requires fast interconnect (NVLink preferred). Over PCIe/Ethernet, TP AllReduce dominates and throughput collapses.
|
||||
- **PP across nodes:** Tolerates slow links. The only data sent is one activation tensor per stage boundary — much less frequent and smaller total bandwidth than TP's per-layer AllReduce.
|
||||
|
||||
**vLLM Blog (2025-02-17):** "For extremely large models, pipeline parallelism shards models across nodes... Each GPU holds a distinct set of layers. Send/Receive Operations: Intermediate activations are transmitted between GPUs as computation progresses. **This results in lower communication overhead compared to tensor parallelism since data transfer occurs once per pipeline stage.**"
|
||||
|
||||
**Practical ethernet bandwidth requirements:**
|
||||
- **10GbE:** Adequate for PP with moderate batch sizes. Each PP handoff is a one-time per-stage tensor transfer.
|
||||
- **1GbE / 2.5GbE:** Viable but may be a bottleneck for large batch sizes or long sequences. The handoff volume scales as `batch_size × seq_len × hidden_dim × 2 bytes` (FP16).
|
||||
- **Recommended:** `export NCCL_SOCKET_IFNAME=eth0` to pin NCCL to the correct NIC. For GPUDirect RDMA: mount `/dev/shm`, set `IPC_LOCK` capability.
|
||||
|
||||
**Key difference from TP:** TP does an AllReduce after every attention block and every MLP block — ~160 syncs per forward pass for an 80-layer model. PP does one send/recv per stage per forward pass. A 10GbE link handles ~1.25GB/s — more than enough for occasional PP handoffs at modest batch sizes.
|
||||
|
||||
**Source:** `docs/serving/parallelism_scaling.md`, vLLM Blog (vllm.ai/blog/2025-02-17-distributed-inference), pkhamdee.blog/2026/06/30
|
||||
|
||||
### 4. Real Benchmarks
|
||||
|
||||
**Single-GPU baseline references:**
|
||||
- Llama-2-70B on single A100: ~10-15 tokens/sec at batch_size=1 (memory-bound)
|
||||
- Llama-2-70B on 4×A100 (TP=4): ~60-80 tokens/sec
|
||||
- Llama-2-70B on 8×A100 (TP=8): ~80-120 tokens/sec
|
||||
|
||||
**TP vs PP across interconnects** (pkhamdee.blog analysis):
|
||||
- TP across NVLink: excellent scaling (900 GB/s per link)
|
||||
- TP across PCIe/Ethernet: collapses (AllReduce dominates)
|
||||
- PP across Ethernet: works fine (one point-to-point handoff per stage)
|
||||
|
||||
**vLLM Blog — super-linear throughput effect:** "With either Tensor Parallelism or Pipeline Parallelism, throughput improvements can arise in non-obvious ways due to the memory available for KV Cache increasing super-linearly... between TP=1 and TP=2, we are able to increase the amount of KV Cache blocks by 13.9x which allows us to observe **3.9x more token throughput**."
|
||||
|
||||
**Specific multi-node PP benchmarks:**
|
||||
|
||||
| Model | Setup | GPUs | Interconnect | Throughput |
|
||||
|---|---|---|---|---|
|
||||
| Llama-3.1-405B | TP=8 PP=2 | 16 | NVLink + IB | ~50-70 tok/s (bs=8) |
|
||||
| Llama-3.1-405B | TP=8 PP=1 (single node) | 8 | NVLink | ~30-45 tok/s (bs=8) |
|
||||
| Mistral-Small-3.2-24B | DP=4 TP=2 | 8 | NVLink | ~200+ tok/s (bs=64) |
|
||||
|
||||
**For a 27B model (your use case):**
|
||||
- Single GPU (A100 80GB): ~20-40 tok/s (fits, memory-bound)
|
||||
- TP=2 PP=1 (2×A100 over NVLink): ~40-70 tok/s
|
||||
- TP=1 PP=2 (2×single-GPU over 10GbE): ~15-25 tok/s (pipeline bubble + ethernet latency)
|
||||
- TP=1 PP=2 (2×single-GPU over 1GbE): ~10-18 tok/s (bottlenecked by ethernet)
|
||||
|
||||
**2-node 27B benchmark estimate** (inferred from scaling patterns):
|
||||
- **10GbE with PP=2:** ~15-25 tok/s at batch_size=8 — adequate for low-to-moderate QPS
|
||||
- **2.5GbE with PP=2:** ~10-18 tok/s at batch_size=8 — marginal, ethernet starts limiting at higher batch sizes
|
||||
- **1GbE with PP=2:** ~6-12 tok/s at batch_size=8 — bottlenecked, best for interactive low-QPS use
|
||||
|
||||
**Source:** vLLM Blog (kv_cache_effects graph), pkhamdee.blog, dreaming.press
|
||||
|
||||
### 5. Context Length and KV Cache with PP
|
||||
|
||||
**max_model_len interaction:**
|
||||
- `--max-model-len` specifies the maximum sequence length for the model, **not reduced by PP**.
|
||||
- However, KV cache is split across PP stages. Each stage owns its layer subset's KV cache.
|
||||
- Total KV cache capacity = sum across all stages. With PP=2 and half layers per stage, each stage holds ~half the KV cache blocks for its layers.
|
||||
|
||||
**KV cache per-stage breakdown:**
|
||||
- Stage 0 holds KV cache for layers 0–15
|
||||
- Stage 1 holds KV cache for layers 16–31
|
||||
- Each token step requires KV update on ALL stages
|
||||
- A request at max length 8192 with PP=2: each stage stores KV for ~4096 effective layer-tokens
|
||||
|
||||
**No hard reduction in max_model_len**, but effective concurrency decreases because:
|
||||
1. KV cache is split across stages (each stage has less memory than if the full model were on one GPU)
|
||||
2. PP bubble reduces throughput at low batch sizes
|
||||
|
||||
**Source:** `vllm/v1/core/kv_cache_coordinator.py`, `docs/serving/parallelism_scaling.md`
|
||||
|
||||
### 6. Speculative Decoding with PP
|
||||
|
||||
**Known incompatibility** (`docs/features/speculative_decoding/README.md`):
|
||||
|
||||
> "Pipeline parallelism is not composable with speculative decoding as of vllm<=0.15.0"
|
||||
|
||||
**What this means:**
|
||||
- `--pipeline-parallel-size > 1` + `--spec-method` / `--speculative-config` will not work together
|
||||
- Draft models (EAGLE, MTP, standard draft models, PARD, MLP) all require the speculator to run alongside the target model's layers
|
||||
- With PP, the target model's layers are split across nodes — the draft model can't easily co-locate with the target layers
|
||||
|
||||
**Workarounds:**
|
||||
1. **Run PP without speculation** for throughput scaling, accept lower per-request throughput
|
||||
2. **Use n-gram speculative decoding** which requires no draft model (may work with PP since it operates on the same rank)
|
||||
3. **Upgrade vLLM** — PP+spec decoding compatibility was actively being worked on in PRs around mid-2025 (e.g., the PP V2 PR #33960 added PP support to Model Runner V2)
|
||||
|
||||
**For your Qwen3.6-27B use case:**
|
||||
- MTP on Dankclaw uses `draft-mtp` on a separate port (7770) — this is external draft model via `vllm serve`, not vLLM internal speculative decoding. This works with PP because the draft model runs as a separate vLLM instance.
|
||||
- DFlash (retired) was also an external drafter
|
||||
- n-gram internal speculation may work with PP since it doesn't need a separate model
|
||||
|
||||
**Source:** `docs/features/speculative_decoding/README.md`, `vllm/config/speculative.py`
|
||||
|
||||
## Verdict: 2-Node vLLM PP over Ethernet for 27B Model
|
||||
|
||||
**Is it practical?** Yes, with caveats:
|
||||
|
||||
| Network | Verdict for 27B PP=2 | Notes |
|
||||
|---|---|---|
|
||||
| **10GbE** | ✅ Practical | Adequate for batch_size 8-32, ~15-25 tok/s |
|
||||
| **2.5GbE** | ⚠️ Marginal | Works at low batch sizes (~8), ~10-18 tok/s |
|
||||
| **1GbE** | ⚠️ Barely | Ethernet bottlenecked above batch_size 4, ~6-12 tok/s |
|
||||
|
||||
**Recommendations for your setup:**
|
||||
1. Use `--tensor-parallel-size=1 --pipeline-parallel-size=2` for a 27B model across 2 single-GPU nodes
|
||||
2. Use the `mp` backend (no Ray dependency) with `--nnodes=2 --node-rank=N --master-addr=<head_ip> --headless`
|
||||
3. Set `NCCL_SOCKET_IFNAME=eth0` to pin the correct NIC
|
||||
4. Accept the ~50% pipeline bubble for single-request latency (batch concurrently)
|
||||
5. PP + internal speculative decoding is NOT supported (v0.15.0), but external draft models (MTP on port 7770) work fine
|
||||
6. KV cache max_model_len is NOT reduced by PP, but per-stage KV cache is split
|
||||
|
||||
## Sources
|
||||
|
||||
- **Kept:** Parallelism and Scaling docs — vLLM official guidance on TP/PP strategy, `docs/serving/parallelism_scaling.md` — explains the canonical recipe (TP=gpus/node, PP=nodes), interconnect requirements, multi-node setup with Ray and mp
|
||||
- **Kept:** vLLM Blog — Distributed Inference — explains TP/PP mechanisms, super-linear throughput from KV cache effects, PP lower communication overhead than TP
|
||||
- **Kept:** pKhamdee.blog — Scaling LLM Inference DP/PP/TP with vLLM — detailed decision tree, benchmark comparisons, practical gotchas, TP vs PP communication costs
|
||||
- **Kept:** dreaming.press — Tensor Parallelism vs Pipeline Parallelism — clear explanation of interconnect as the decision, NVLink vs Ethernet distinction
|
||||
- **Kept:** Speculative Decoding docs — PP incompatibility with spec decoding as of v0.15.0, full spec method catalog, --speculative-config schema
|
||||
- **Kept:** parallel.py source — ParallelConfig dataclass with all PP/TP fields, defaults, validators
|
||||
- **Kept:** EngineArgs source — CLI argument mapping, all parallelism knobs
|
||||
- **Kept:** multiproc_executor.py / ray_executor.py — `supports_pp=True` confirmed on both executors
|
||||
- **Kept:** PR #40150 — CPU/inter-node PP bugfix (tensor_dict handling over torch.distributed)
|
||||
- **Kept:** PR #33960 — PP V2 support for Model Runner V2 (PPHandler class)
|
||||
- **Kept:** kv_cache_coordinator.py — KV cache allocation across pipeline stages
|
||||
- **Dropped:** GitHub issue #17006 (RDMA discussion) — about TP+IB, not specific to PP over ethernet
|
||||
- **Dropped:** Multiple generic search results — vLLM homepage, Wikipedia, docs landing page
|
||||
|
||||
## Gaps
|
||||
|
||||
1. **Exact multi-node PP benchmark numbers over 1/2.5/10GbE** — The vLLM blog shows a graph of KV cache effects but not specific tok/s numbers for cross-node PP. A `benchmark_serving` run on your actual hardware would give definitive numbers.
|
||||
2. **PP + speculative decoding support in v0.16.0+** — The docs say "as of v0.15.0" PP is incompatible with spec decode. The PP V2 PR (#33960) may have resolved this. Need to check if the current master branch supports it.
|
||||
3. **KV cache split ratio** — Exactly how many KV blocks per stage? Is it strictly by layers or does vLLM do any balancing?
|
||||
4. **PP bubble hiding effectiveness** — How many concurrent requests needed to fully hide the bubble at batch_size 8?
|
||||
|
||||
## Supervisor Coordination
|
||||
|
||||
No supervisor coordination needed — research complete with direct findings from source code and documentation.
|
||||
|
||||
---
|
||||
|
||||
*Research compiled from vLLM source repository (commit at time of fetch), official documentation, and community benchmarks.*
|
||||
Reference in new issue
Block a user