Frank
52dbe18486
`scripts/port-census.py` built its corpus from the new directory, so each page under test was held by itself: every era page reported 0 orphan lines. With the corpus excluding the file under test, each candidate holds 14-63 sole-record lines, so no era page is padding and none was trashed. Deletions of restatement only: - Multi-Node Runtimes.md (-257): restated launch material owned by Qwen3.8 PP Deployment.md; index rows and two Related bullets repointed. - log.md: 66 headings folded into 35 dated entries (one entry per date, newest first), titled headings kept as `date — Title` per the spec's wording, and 78 lines of transient deploy chatter dropped where every fact token in the line is held by an owner page (preserved in .AGENTTRASH/20261019-log-transient/). - Authentic: the 5-page split reverted (-93); the parent carries `okf-exempt: length` and the three stub pages stay deleted. check-okf.py accepts `## YYYY-MM-DD — Title`; port-census.py excludes the file under test and gained a docstring note about the trap. One-off compaction tool moved to ~/.AGENTTRASH. Full suite: 451 tests pass, 0 OKF violations, 155 pages, 0 unresolved links.
72 lines
2.8 KiB
Markdown
72 lines
2.8 KiB
Markdown
# Pipeline Parallelism
|
|
|
|
This section covers running one large model across two machines. It focuses
|
|
on pipeline parallelism (layer split) over ethernet. The research answers
|
|
what speeds are possible and which tools work. The sample scripts show a
|
|
two-node Qwen3.6-27B Q8 split for reading only.
|
|
|
|
## Terminology
|
|
|
|
Pipeline parallelism (PP) splits model layers across machines. Each machine
|
|
computes its layers and passes hidden states to the next machine. Tensor
|
|
parallelism (TP) splits each layer across machines and requires an
|
|
all-reduce per layer. PP is the only realistic option over ethernet.
|
|
|
|
Speculative decoding is a separate feature. It uses a small draft model (or
|
|
n-grams) to propose tokens that the main model verifies in one pass. It can
|
|
run together with PP in llama.cpp.
|
|
|
|
## The Two Runtimes
|
|
|
|
```mermaid
|
|
graph LR
|
|
subgraph llamaRPC[llama.cpp RPC]
|
|
A1[ggml-rpc-server on node B] --> A2[llama-server on node A<br/>--rpc nodeB:50052]
|
|
end
|
|
subgraph vllmPP[vLLM PP]
|
|
B1[vllm serve node 0<br/>tp=1 pp=2 rank 0] <-->|hidden states| B2[vllm serve node 1<br/>tp=1 pp=2 rank 1]
|
|
end
|
|
```
|
|
|
|
heterogeneous nodes over ethernet.
|
|
multiprocessing and does not compose with speculative decoding yet.
|
|
|
|
## What the Research Found
|
|
|
|
- [[Ethernet Bandwidth Math]] — Bytes-per-token math for pipeline versus tensor parallelism, and the link budgets that follow from it.
|
|
(about 10 KiB per token for a 27B model). 1 GbE supports hundreds of
|
|
tokens per second on paper; the real limit is latency and RPC overhead.
|
|
works today via llama.cpp RPC with separate CUDA and ROCm builds. The
|
|
exact NVIDIA-desktop + Strix-Halo-NUC pair has not been benchmarked with
|
|
published numbers yet.
|
|
methods (MTP, DFlash, ngram-mod) work with RPC. DSpark is vLLM-only.
|
|
- [[Expected Performance]] — Estimated split-model throughput over ethernet, with the benchmark anchors the estimates rest on.
|
|
across an RTX 3090 and a Strix Halo NUC over 1/2.5/10 GbE.
|
|
|
|
## Deployment
|
|
|
|
- [[Qwen3.8 PP Deployment]] — The real split deployment: which node is head and which is worker, the flags, and the verified gotchas.
|
|
(Strix), worker on Dankserver (RTX 3090), b10433 on both sides, scripts in
|
|
`scripts/`, morning-of test runbook. Verified gotchas: no `--device` with
|
|
`--rpc`, default split is proportional to reported free memory (pin 1,1),
|
|
the iGPU side defaults to the ROCm backend.
|
|
|
|
## Sample Scripts
|
|
|
|
- [[Sample Script Set]] — An educational two-node RPC split script set, for reading only — never run it or deploy it.
|
|
Qwen3.6-27B-Q8_K_P with MTP, DFlash, and ngram-mod variants, sized for
|
|
the largest context that fits. For educational reading only. Do not run
|
|
it in production.
|
|
|
|
## See Also
|
|
|
|
- [[DFlash DSpark Speculative Decoding]]
|
|
- [[frank-zealot]]
|
|
- [[Dankserver System Specs]]
|
|
- [[Dankclaw System Specs]]
|
|
|
|
## Pages
|
|
|
|
| Page | Owns |
|
|
|---|---|
|