Frank
52dbe18486
`scripts/port-census.py` built its corpus from the new directory, so each page under test was held by itself: every era page reported 0 orphan lines. With the corpus excluding the file under test, each candidate holds 14-63 sole-record lines, so no era page is padding and none was trashed. Deletions of restatement only: - Multi-Node Runtimes.md (-257): restated launch material owned by Qwen3.8 PP Deployment.md; index rows and two Related bullets repointed. - log.md: 66 headings folded into 35 dated entries (one entry per date, newest first), titled headings kept as `date — Title` per the spec's wording, and 78 lines of transient deploy chatter dropped where every fact token in the line is held by an owner page (preserved in .AGENTTRASH/20261019-log-transient/). - Authentic: the 5-page split reverted (-93); the parent carries `okf-exempt: length` and the three stub pages stay deleted. check-okf.py accepts `## YYYY-MM-DD — Title`; port-census.py excludes the file under test and gained a docstring note about the trap. One-off compaction tool moved to ~/.AGENTTRASH. Full suite: 451 tests pass, 0 OKF violations, 155 pages, 0 unresolved links.
2.8 KiB
Pipeline Parallelism
This section covers running one large model across two machines. It focuses on pipeline parallelism (layer split) over ethernet. The research answers what speeds are possible and which tools work. The sample scripts show a two-node Qwen3.6-27B Q8 split for reading only.
Terminology
Pipeline parallelism (PP) splits model layers across machines. Each machine computes its layers and passes hidden states to the next machine. Tensor parallelism (TP) splits each layer across machines and requires an all-reduce per layer. PP is the only realistic option over ethernet.
Speculative decoding is a separate feature. It uses a small draft model (or n-grams) to propose tokens that the main model verifies in one pass. It can run together with PP in llama.cpp.
The Two Runtimes
graph LR
subgraph llamaRPC[llama.cpp RPC]
A1[ggml-rpc-server on node B] --> A2[llama-server on node A<br/>--rpc nodeB:50052]
end
subgraph vllmPP[vLLM PP]
B1[vllm serve node 0<br/>tp=1 pp=2 rank 0] <-->|hidden states| B2[vllm serve node 1<br/>tp=1 pp=2 rank 1]
end
heterogeneous nodes over ethernet. multiprocessing and does not compose with speculative decoding yet.
What the Research Found
- Ethernet Bandwidth Math — Bytes-per-token math for pipeline versus tensor parallelism, and the link budgets that follow from it. (about 10 KiB per token for a 27B model). 1 GbE supports hundreds of tokens per second on paper; the real limit is latency and RPC overhead. works today via llama.cpp RPC with separate CUDA and ROCm builds. The exact NVIDIA-desktop + Strix-Halo-NUC pair has not been benchmarked with published numbers yet. methods (MTP, DFlash, ngram-mod) work with RPC. DSpark is vLLM-only.
- Expected Performance — Estimated split-model throughput over ethernet, with the benchmark anchors the estimates rest on. across an RTX 3090 and a Strix Halo NUC over 1/2.5/10 GbE.
Deployment
- Qwen3.8 PP Deployment — The real split deployment: which node is head and which is worker, the flags, and the verified gotchas.
(Strix), worker on Dankserver (RTX 3090), b10433 on both sides, scripts in
scripts/, morning-of test runbook. Verified gotchas: no--devicewith--rpc, default split is proportional to reported free memory (pin 1,1), the iGPU side defaults to the ROCm backend.
Sample Scripts
- Sample Script Set — An educational two-node RPC split script set, for reading only — never run it or deploy it. Qwen3.6-27B-Q8_K_P with MTP, DFlash, and ngram-mod variants, sized for the largest context that fits. For educational reading only. Do not run it in production.
See Also
Pages
| Page | Owns |
|---|