Frank
a5fe045b58
Structural repair: - 13 pages were the same document twice (each exactly 2x its HEAD size), a second frontmatter block whose closing fence had been glued onto a body line by a frontmatter rewrite that re-read its own output. Un-doubled by cutting at the second frontmatter and keeping the copy whose body is a superset; the orphan census reports 0 lost tokens against HEAD. - log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first — 66 dated headings merged into 34, titles demoted to bold leads, every content line verified present. The file had been descending at the head and ascending at the tail. Ownership moves: - New `LLM/frank-medium.md` is the locator for the retired Dankserver endpoint: era windows quoted from each era page's own banner, no numbers restated, and the two unresolved date conflicts recorded instead of silently resolved. `frank-archon.md` links to it instead of restating the pre-rename window. - The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone; `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor verdicts and cross-split spec decode, and the four links that pointed at the stubs were repointed at it or at the drafter page. - Authentik's three stub pages fold into `Authentik.md`, with the process detail and the outpost reference split out so the parent stays under the length hint. Verified corrections (host census + live probe, 2026-10-19): - The MXFP4 checkpoint two pages called the "on-disk rollback" is gone: /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on the era page that owns that payload; the swap page now links to it. - frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2 arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run census is kept once (155 requests) instead of twice (~120 and 155). - Fleet shapes re-confirmed against the live endpoints before touching any page: 3x200000 fp8 KV, 2x200704, 2x262144. Index rows are regenerated, not hand-carved. Net -1,849 vault lines.
73 lines
2.9 KiB
Markdown
73 lines
2.9 KiB
Markdown
# Pipeline Parallelism
|
|
|
|
This section covers running one large model across two machines. It focuses
|
|
on pipeline parallelism (layer split) over ethernet. The research answers
|
|
what speeds are possible and which tools work. The sample scripts show a
|
|
two-node Qwen3.6-27B Q8 split for reading only.
|
|
|
|
## Terminology
|
|
|
|
Pipeline parallelism (PP) splits model layers across machines. Each machine
|
|
computes its layers and passes hidden states to the next machine. Tensor
|
|
parallelism (TP) splits each layer across machines and requires an
|
|
all-reduce per layer. PP is the only realistic option over ethernet.
|
|
|
|
Speculative decoding is a separate feature. It uses a small draft model (or
|
|
n-grams) to propose tokens that the main model verifies in one pass. It can
|
|
run together with PP in llama.cpp.
|
|
|
|
## The Two Runtimes
|
|
|
|
```mermaid
|
|
graph LR
|
|
subgraph llamaRPC[llama.cpp RPC]
|
|
A1[ggml-rpc-server on node B] --> A2[llama-server on node A<br/>--rpc nodeB:50052]
|
|
end
|
|
subgraph vllmPP[vLLM PP]
|
|
B1[vllm serve node 0<br/>tp=1 pp=2 rank 0] <-->|hidden states| B2[vllm serve node 1<br/>tp=1 pp=2 rank 1]
|
|
end
|
|
```
|
|
|
|
heterogeneous nodes over ethernet.
|
|
multiprocessing and does not compose with speculative decoding yet.
|
|
|
|
## What the Research Found
|
|
|
|
- [[Ethernet Bandwidth Math]] — Bytes-per-token math for pipeline versus tensor parallelism, and the link budgets that follow from it.
|
|
(about 10 KiB per token for a 27B model). 1 GbE supports hundreds of
|
|
tokens per second on paper; the real limit is latency and RPC overhead.
|
|
works today via llama.cpp RPC with separate CUDA and ROCm builds. The
|
|
exact NVIDIA-desktop + Strix-Halo-NUC pair has not been benchmarked with
|
|
published numbers yet.
|
|
methods (MTP, DFlash, ngram-mod) work with RPC. DSpark is vLLM-only.
|
|
- [[Expected Performance]] — Estimated split-model throughput over ethernet, with the benchmark anchors the estimates rest on.
|
|
across an RTX 3090 and a Strix Halo NUC over 1/2.5/10 GbE.
|
|
|
|
## Deployment
|
|
|
|
- [[Qwen3.8 PP Deployment]] — The real split deployment: which node is head and which is worker, the flags, and the verified gotchas.
|
|
(Strix), worker on Dankserver (RTX 3090), b10433 on both sides, scripts in
|
|
`scripts/`, morning-of test runbook. Verified gotchas: no `--device` with
|
|
`--rpc`, default split is proportional to reported free memory (pin 1,1),
|
|
the iGPU side defaults to the ROCm backend.
|
|
|
|
## Sample Scripts
|
|
|
|
- [[Sample Script Set]] — An educational two-node RPC split script set, for reading only — never run it or deploy it.
|
|
Qwen3.6-27B-Q8_K_P with MTP, DFlash, and ngram-mod variants, sized for
|
|
the largest context that fits. For educational reading only. Do not run
|
|
it in production.
|
|
|
|
## See Also
|
|
|
|
- [[DFlash DSpark Speculative Decoding]]
|
|
- [[frank-zealot]]
|
|
- [[Dankserver System Specs]]
|
|
- [[Dankclaw System Specs]]
|
|
|
|
## Pages
|
|
|
|
| Page | Owns |
|
|
|---|---|
|
|
| [[Multi-Node Runtimes]] | How each two-machine runtime is built and launched, what it can combine with, and the mixed-vendor verdicts. |
|