Frank
017d3704d2
The LLM fleet documentation (deployment runbook, model runbooks, benchmarks, research archive, retired-topology history) joins the project knowledge base as docs/LLM/. Wikilinks resolve by filename, so no link updates are needed for the move.
3.1 KiB
3.1 KiB
type, title, description, tags, timestamp
| type | title | description | tags | timestamp | |||
|---|---|---|---|---|---|---|---|
| index | Pipeline Parallelism | Multi-node LLM inference research and sample scripts for llama.cpp RPC and vLLM PP over ethernet. |
|
2026-08-15 |
Pipeline Parallelism
This section covers running one large model across two machines. It focuses on pipeline parallelism (layer split) over ethernet. The research answers what speeds are possible and which tools work. The sample scripts show a two-node Qwen3.6-27B Q8 split for reading only.
Terminology
Pipeline parallelism (PP) splits model layers across machines. Each machine computes its layers and passes hidden states to the next machine. Tensor parallelism (TP) splits each layer across machines and requires an all-reduce per layer. PP is the only realistic option over ethernet.
Speculative decoding is a separate feature. It uses a small draft model (or n-grams) to propose tokens that the main model verifies in one pass. It can run together with PP in llama.cpp.
The Two Runtimes
graph LR
subgraph llamaRPC[llama.cpp RPC]
A1[ggml-rpc-server on node B] --> A2[llama-server on node A<br/>--rpc nodeB:50052]
end
subgraph vllmPP[vLLM PP]
B1[vllm serve node 0<br/>tp=1 pp=2 rank 0] <-->|hidden states| B2[vllm serve node 1<br/>tp=1 pp=2 rank 1]
end
- llama.cpp RPC — the working, production-capable path for two heterogeneous nodes over ethernet.
- vLLM Pipeline Parallelism — supported but heavier; needs Ray or multiprocessing and does not compose with speculative decoding yet.
What the Research Found
- Ethernet Bandwidth Math — a PP boundary moves only the hidden state (about 10 KiB per token for a 27B model). 1 GbE supports hundreds of tokens per second on paper; the real limit is latency and RPC overhead.
- Mixed-Vendor Experiments — NVIDIA + Strix Halo multi-node inference works today via llama.cpp RPC with separate CUDA and ROCm builds. The exact NVIDIA-desktop + Strix-Halo-NUC pair has not been benchmarked with published numbers yet.
- Qwen3.6 27B Spec Methods — Q8 GGUF variants, and which speculative methods (MTP, DFlash, ngram-mod) work with RPC. DSpark is vLLM-only.
- Expected Performance — estimated tokens per second for a 27B Q8 split across an RTX 3090 and a Strix Halo NUC over 1/2.5/10 GbE.
Deployment
- Qwen3.8 PP Deployment — the real Qwen3.8-27B split: head on Dankclaw
(Strix), worker on Dankserver (RTX 3090), b10433 on both sides, scripts in
scripts/, morning-of test runbook. Verified gotchas: no--devicewith--rpc, default split is proportional to reported free memory (pin 1,1), the iGPU side defaults to the ROCm backend.
Sample Scripts
- Sample Script Set — a two-node llama.cpp RPC split of Qwen3.6-27B-Q8_K_P with MTP, DFlash, and ngram-mod variants, sized for the largest context that fits. For educational reading only. Do not run it in production.