Frank
8212d4451a
Mechanical identity rename across the knowledge base, AGENTS.md, README, the
hermes config example, and homepage services.json. No behavior change; served
aliases on the LLM endpoints change with the deploy of commit 69f87c8.
82 lines
3.1 KiB
Markdown
82 lines
3.1 KiB
Markdown
---
|
|
type: index
|
|
title: Pipeline Parallelism
|
|
description: Multi-node LLM inference research and sample scripts for llama.cpp RPC and vLLM PP over ethernet.
|
|
tags:
|
|
- pipeline-parallelism
|
|
- distributed
|
|
- index
|
|
timestamp: 2026-08-15
|
|
---
|
|
|
|
# Pipeline Parallelism
|
|
|
|
This section covers running one large model across two machines. It focuses
|
|
on pipeline parallelism (layer split) over ethernet. The research answers
|
|
what speeds are possible and which tools work. The sample scripts show a
|
|
two-node Qwen3.6-27B Q8 split for reading only.
|
|
|
|
## Terminology
|
|
|
|
Pipeline parallelism (PP) splits model layers across machines. Each machine
|
|
computes its layers and passes hidden states to the next machine. Tensor
|
|
parallelism (TP) splits each layer across machines and requires an
|
|
all-reduce per layer. PP is the only realistic option over ethernet.
|
|
|
|
Speculative decoding is a separate feature. It uses a small draft model (or
|
|
n-grams) to propose tokens that the main model verifies in one pass. It can
|
|
run together with PP in llama.cpp.
|
|
|
|
## The Two Runtimes
|
|
|
|
```mermaid
|
|
graph LR
|
|
subgraph llamaRPC[llama.cpp RPC]
|
|
A1[ggml-rpc-server on node B] --> A2[llama-server on node A<br/>--rpc nodeB:50052]
|
|
end
|
|
subgraph vllmPP[vLLM PP]
|
|
B1[vllm serve node 0<br/>tp=1 pp=2 rank 0] <-->|hidden states| B2[vllm serve node 1<br/>tp=1 pp=2 rank 1]
|
|
end
|
|
```
|
|
|
|
- [[llama.cpp RPC]] — the working, production-capable path for two
|
|
heterogeneous nodes over ethernet.
|
|
- [[vLLM Pipeline Parallelism]] — supported but heavier; needs Ray or
|
|
multiprocessing and does not compose with speculative decoding yet.
|
|
|
|
## What the Research Found
|
|
|
|
- [[Ethernet Bandwidth Math]] — a PP boundary moves only the hidden state
|
|
(about 10 KiB per token for a 27B model). 1 GbE supports hundreds of
|
|
tokens per second on paper; the real limit is latency and RPC overhead.
|
|
- [[Mixed-Vendor Experiments]] — NVIDIA + Strix Halo multi-node inference
|
|
works today via llama.cpp RPC with separate CUDA and ROCm builds. The
|
|
exact NVIDIA-desktop + Strix-Halo-NUC pair has not been benchmarked with
|
|
published numbers yet.
|
|
- [[Qwen3.6 27B Spec Methods]] — Q8 GGUF variants, and which speculative
|
|
methods (MTP, DFlash, ngram-mod) work with RPC. DSpark is vLLM-only.
|
|
- [[Expected Performance]] — estimated tokens per second for a 27B Q8 split
|
|
across an RTX 3090 and a Strix Halo NUC over 1/2.5/10 GbE.
|
|
|
|
## Deployment
|
|
|
|
- [[Qwen3.8 PP Deployment]] — the real Qwen3.8-27B split: head on Dankclaw
|
|
(Strix), worker on Dankserver (RTX 3090), b10433 on both sides, scripts in
|
|
`scripts/`, morning-of test runbook. Verified gotchas: no `--device` with
|
|
`--rpc`, default split is proportional to reported free memory (pin 1,1),
|
|
the iGPU side defaults to the ROCm backend.
|
|
|
|
## Sample Scripts
|
|
|
|
- [[Sample Script Set]] — a two-node llama.cpp RPC split of
|
|
Qwen3.6-27B-Q8_K_P with MTP, DFlash, and ngram-mod variants, sized for
|
|
the largest context that fits. For educational reading only. Do not run
|
|
it in production.
|
|
|
|
## See Also
|
|
|
|
- [[DFlash DSpark Speculative Decoding]]
|
|
- [[frank-small-benchmark]]
|
|
- [[Dankserver System Specs]]
|
|
- [[Dankclaw System Specs]]
|