Frank
219a097b39
Every payload swap a live endpoint has ever run is now one row in that endpoint's Eras table, with its rollback path in the same row. No per-attempt page, no per-swap ledger, nowhere else. - Added: LLM/frank-archon.md (radiance, 2x R9700 TP=2), LLM/frank-dragoon.md (halogen flash-server, 19 eras), LLM/frank-zealot.md (llama.cpp PrismML fork on the 3090). Each carries Current shape, Eras, Laws, wire contract, files. - Deleted: 45 per-payload runbooks -- the frank-dragoon line (11 including the pre-name halogen/kingjones/kairic/CIRU/eemin/agentionai/dflash2 pages and the benchmark page), the frank-zealot line (llama.cpp/vLLM/ik_llama/SGLang eras), the frank-archon line (vLLM/FreeToken/R9V/llama.cpp/radiance eras), plus frank-medium, frank-npu and the ubatch/PCIe one-offs that owned nothing. - Rewired: every wikilink and markdown-path reference to a folded page points at the endpoint page instead; the three Research see-also lists and the ubatch sweep's runbook link included. No anchored links existed in the set. - Regenerated both index pages as pure locators -- one row, one clause, no numbers -- so the docs_completeness locator law holds for all 153 pages. tests/docs_completeness.bats green: index links every page, no unresolved links. The pages moved to .AGENTTRASH/20261018-doc-dedupe/LLM/ (gitignored, and git history holds every byte).
71 lines
2.8 KiB
Markdown
71 lines
2.8 KiB
Markdown
# Pipeline Parallelism
|
|
|
|
This section covers running one large model across two machines. It focuses
|
|
on pipeline parallelism (layer split) over ethernet. The research answers
|
|
what speeds are possible and which tools work. The sample scripts show a
|
|
two-node Qwen3.6-27B Q8 split for reading only.
|
|
|
|
## Terminology
|
|
|
|
Pipeline parallelism (PP) splits model layers across machines. Each machine
|
|
computes its layers and passes hidden states to the next machine. Tensor
|
|
parallelism (TP) splits each layer across machines and requires an
|
|
all-reduce per layer. PP is the only realistic option over ethernet.
|
|
|
|
Speculative decoding is a separate feature. It uses a small draft model (or
|
|
n-grams) to propose tokens that the main model verifies in one pass. It can
|
|
run together with PP in llama.cpp.
|
|
|
|
## The Two Runtimes
|
|
|
|
```mermaid
|
|
graph LR
|
|
subgraph llamaRPC[llama.cpp RPC]
|
|
A1[ggml-rpc-server on node B] --> A2[llama-server on node A<br/>--rpc nodeB:50052]
|
|
end
|
|
subgraph vllmPP[vLLM PP]
|
|
B1[vllm serve node 0<br/>tp=1 pp=2 rank 0] <-->|hidden states| B2[vllm serve node 1<br/>tp=1 pp=2 rank 1]
|
|
end
|
|
```
|
|
|
|
- [[llama.cpp RPC]] — the working, production-capable path for two
|
|
heterogeneous nodes over ethernet.
|
|
- [[vLLM Pipeline Parallelism]] — supported but heavier; needs Ray or
|
|
multiprocessing and does not compose with speculative decoding yet.
|
|
|
|
## What the Research Found
|
|
|
|
- [[Ethernet Bandwidth Math]] — a PP boundary moves only the hidden state
|
|
(about 10 KiB per token for a 27B model). 1 GbE supports hundreds of
|
|
tokens per second on paper; the real limit is latency and RPC overhead.
|
|
- [[Mixed-Vendor Experiments]] — NVIDIA + Strix Halo multi-node inference
|
|
works today via llama.cpp RPC with separate CUDA and ROCm builds. The
|
|
exact NVIDIA-desktop + Strix-Halo-NUC pair has not been benchmarked with
|
|
published numbers yet.
|
|
- [[Qwen3.6 27B Spec Methods]] — Q8 GGUF variants, and which speculative
|
|
methods (MTP, DFlash, ngram-mod) work with RPC. DSpark is vLLM-only.
|
|
- [[Expected Performance]] — estimated tokens per second for a 27B Q8 split
|
|
across an RTX 3090 and a Strix Halo NUC over 1/2.5/10 GbE.
|
|
|
|
## Deployment
|
|
|
|
- [[Qwen3.8 PP Deployment]] — the real Qwen3.8-27B split: head on Dankclaw
|
|
(Strix), worker on Dankserver (RTX 3090), b10433 on both sides, scripts in
|
|
`scripts/`, morning-of test runbook. Verified gotchas: no `--device` with
|
|
`--rpc`, default split is proportional to reported free memory (pin 1,1),
|
|
the iGPU side defaults to the ROCm backend.
|
|
|
|
## Sample Scripts
|
|
|
|
- [[Sample Script Set]] — a two-node llama.cpp RPC split of
|
|
Qwen3.6-27B-Q8_K_P with MTP, DFlash, and ngram-mod variants, sized for
|
|
the largest context that fits. For educational reading only. Do not run
|
|
it in production.
|
|
|
|
## See Also
|
|
|
|
- [[DFlash DSpark Speculative Decoding]]
|
|
- [[frank-zealot]]
|
|
- [[Dankserver System Specs]]
|
|
- [[Dankclaw System Specs]]
|