Frank
93d1181fee
- Migrate all 127 non-reserved docs to OKF v0.1 frontmatter (type/description/tags/timestamp); fix 13 YAML issues, 2 duplicate-FM blocks, strip reserved-page FM - Restructure log.md to OKF (## YYYY-MM-DD, newest-first) - Fact-check against scripts/AGENTS.md ground truth: fix 6 false 'Active' banners on retired runbooks, 96->114 GiB GTT, deployment.md vs deploy-plan.py/deploy-remote.sh, Deployment Flow frank-medium=Quadlet, Calibre 9.9.0->9.15.0 + CVE-2025-7404 conflation, Copyparty v1.20.24, RomM 5.3.0 breaking change - Move *arr docs to new FrankBook-Pro/ section (git mv, link-safe) - Write active runbook bodies from ground truth (frank-large, frank-small) - Add 3 research briefs + HTML report: Local TTS and Voice Cloning, AI Job Search and Resume Tools, Most Popular Self-Hosted AI Apps (Jun-Sep 2026) - Add upstream-version rows to ComfyUI/OpenWebUI/Hermes app docs - All 336 tests pass; 136 pages, 0 unresolved wikilinks
71 lines
2.8 KiB
Markdown
71 lines
2.8 KiB
Markdown
# Pipeline Parallelism
|
|
|
|
This section covers running one large model across two machines. It focuses
|
|
on pipeline parallelism (layer split) over ethernet. The research answers
|
|
what speeds are possible and which tools work. The sample scripts show a
|
|
two-node Qwen3.6-27B Q8 split for reading only.
|
|
|
|
## Terminology
|
|
|
|
Pipeline parallelism (PP) splits model layers across machines. Each machine
|
|
computes its layers and passes hidden states to the next machine. Tensor
|
|
parallelism (TP) splits each layer across machines and requires an
|
|
all-reduce per layer. PP is the only realistic option over ethernet.
|
|
|
|
Speculative decoding is a separate feature. It uses a small draft model (or
|
|
n-grams) to propose tokens that the main model verifies in one pass. It can
|
|
run together with PP in llama.cpp.
|
|
|
|
## The Two Runtimes
|
|
|
|
```mermaid
|
|
graph LR
|
|
subgraph llamaRPC[llama.cpp RPC]
|
|
A1[ggml-rpc-server on node B] --> A2[llama-server on node A<br/>--rpc nodeB:50052]
|
|
end
|
|
subgraph vllmPP[vLLM PP]
|
|
B1[vllm serve node 0<br/>tp=1 pp=2 rank 0] <-->|hidden states| B2[vllm serve node 1<br/>tp=1 pp=2 rank 1]
|
|
end
|
|
```
|
|
|
|
- [[llama.cpp RPC]] — the working, production-capable path for two
|
|
heterogeneous nodes over ethernet.
|
|
- [[vLLM Pipeline Parallelism]] — supported but heavier; needs Ray or
|
|
multiprocessing and does not compose with speculative decoding yet.
|
|
|
|
## What the Research Found
|
|
|
|
- [[Ethernet Bandwidth Math]] — a PP boundary moves only the hidden state
|
|
(about 10 KiB per token for a 27B model). 1 GbE supports hundreds of
|
|
tokens per second on paper; the real limit is latency and RPC overhead.
|
|
- [[Mixed-Vendor Experiments]] — NVIDIA + Strix Halo multi-node inference
|
|
works today via llama.cpp RPC with separate CUDA and ROCm builds. The
|
|
exact NVIDIA-desktop + Strix-Halo-NUC pair has not been benchmarked with
|
|
published numbers yet.
|
|
- [[Qwen3.6 27B Spec Methods]] — Q8 GGUF variants, and which speculative
|
|
methods (MTP, DFlash, ngram-mod) work with RPC. DSpark is vLLM-only.
|
|
- [[Expected Performance]] — estimated tokens per second for a 27B Q8 split
|
|
across an RTX 3090 and a Strix Halo NUC over 1/2.5/10 GbE.
|
|
|
|
## Deployment
|
|
|
|
- [[Qwen3.8 PP Deployment]] — the real Qwen3.8-27B split: head on Dankclaw
|
|
(Strix), worker on Dankserver (RTX 3090), b10433 on both sides, scripts in
|
|
`scripts/`, morning-of test runbook. Verified gotchas: no `--device` with
|
|
`--rpc`, default split is proportional to reported free memory (pin 1,1),
|
|
the iGPU side defaults to the ROCm backend.
|
|
|
|
## Sample Scripts
|
|
|
|
- [[Sample Script Set]] — a two-node llama.cpp RPC split of
|
|
Qwen3.6-27B-Q8_K_P with MTP, DFlash, and ngram-mod variants, sized for
|
|
the largest context that fits. For educational reading only. Do not run
|
|
it in production.
|
|
|
|
## See Also
|
|
|
|
- [[DFlash DSpark Speculative Decoding]]
|
|
- [[frank-small-benchmark]]
|
|
- [[Dankserver System Specs]]
|
|
- [[Dankclaw System Specs]]
|