docs: pipeline parallelism research + educational sample script set
Compiles the 5-subagent research fan-out (llama.cpp RPC, vLLM PP, bandwidth math, mixed-vendor experiments, Qwen3.6-27B spec methods) into the KB under docs/Pipeline Parallelism/ with an index, expected performance estimates (1/2.5/10 GbE), and an EDUCATIONAL-ONLY two-node Qwen3.6-27B Q8 split script set (MTP/DFlash/ngram-mod variants, DSpark noted as vLLM-only). Nothing here is deployed or intended to run.
This commit is contained in:
1 parent
8d1d9baa82
commit
db69fc8192
9 files changed
+1915
No files matched your search
@@ -0,0 +1,73 @@
|
||||
---
|
||||
type: index
|
||||
title: Pipeline Parallelism
|
||||
description: Multi-node LLM inference research and sample scripts for llama.cpp RPC and vLLM PP over ethernet.
|
||||
tags:
|
||||
- pipeline-parallelism
|
||||
- distributed
|
||||
- index
|
||||
timestamp: 2026-08-14
|
||||
---
|
||||
|
||||
# Pipeline Parallelism
|
||||
|
||||
This section covers running one large model across two machines. It focuses
|
||||
on pipeline parallelism (layer split) over ethernet. The research answers
|
||||
what speeds are possible and which tools work. The sample scripts show a
|
||||
two-node Qwen3.6-27B Q8 split for reading only.
|
||||
|
||||
## Terminology
|
||||
|
||||
Pipeline parallelism (PP) splits model layers across machines. Each machine
|
||||
computes its layers and passes hidden states to the next machine. Tensor
|
||||
parallelism (TP) splits each layer across machines and requires an
|
||||
all-reduce per layer. PP is the only realistic option over ethernet.
|
||||
|
||||
Speculative decoding is a separate feature. It uses a small draft model (or
|
||||
n-grams) to propose tokens that the main model verifies in one pass. It can
|
||||
run together with PP in llama.cpp.
|
||||
|
||||
## The Two Runtimes
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
subgraph llamaRPC[llama.cpp RPC]
|
||||
A1[ggml-rpc-server on node B] --> A2[llama-server on node A<br/>--rpc nodeB:50052]
|
||||
end
|
||||
subgraph vllmPP[vLLM PP]
|
||||
B1[vllm serve node 0<br/>tp=1 pp=2 rank 0] <-->|hidden states| B2[vllm serve node 1<br/>tp=1 pp=2 rank 1]
|
||||
end
|
||||
```
|
||||
|
||||
- [[llama.cpp RPC]] — the working, production-capable path for two
|
||||
heterogeneous nodes over ethernet.
|
||||
- [[vLLM Pipeline Parallelism]] — supported but heavier; needs Ray or
|
||||
multiprocessing and does not compose with speculative decoding yet.
|
||||
|
||||
## What the Research Found
|
||||
|
||||
- [[Ethernet Bandwidth Math]] — a PP boundary moves only the hidden state
|
||||
(about 10 KiB per token for a 27B model). 1 GbE supports hundreds of
|
||||
tokens per second on paper; the real limit is latency and RPC overhead.
|
||||
- [[Mixed-Vendor Experiments]] — NVIDIA + Strix Halo multi-node inference
|
||||
works today via llama.cpp RPC with separate CUDA and ROCm builds. The
|
||||
exact NVIDIA-desktop + Strix-Halo-NUC pair has not been benchmarked with
|
||||
published numbers yet.
|
||||
- [[Qwen3.6 27B Spec Methods]] — Q8 GGUF variants, and which speculative
|
||||
methods (MTP, DFlash, ngram-mod) work with RPC. DSpark is vLLM-only.
|
||||
- [[Expected Performance]] — estimated tokens per second for a 27B Q8 split
|
||||
across an RTX 3090 and a Strix Halo NUC over 1/2.5/10 GbE.
|
||||
|
||||
## Sample Scripts
|
||||
|
||||
- [[Sample Script Set]] — a two-node llama.cpp RPC split of
|
||||
Qwen3.6-27B-Q8_K_P with MTP, DFlash, and ngram-mod variants, sized for
|
||||
the largest context that fits. For educational reading only. Do not run
|
||||
it in production.
|
||||
|
||||
## See Also
|
||||
|
||||
- [[DFlash DSpark Speculative Decoding]]
|
||||
- [[frank-default-small-benchmark]]
|
||||
- [[Dankserver System Specs]]
|
||||
- [[Dankclaw System Specs]]
|
||||
Reference in new issue
Block a user