diff --git a/serverAppConfigs-docs/LLM/Pipeline Parallelism/Expected Performance.md b/serverAppConfigs-docs/LLM/Pipeline Parallelism/Expected Performance.md index 4fc57e2..201b5ca 100644 --- a/serverAppConfigs-docs/LLM/Pipeline Parallelism/Expected Performance.md +++ b/serverAppConfigs-docs/LLM/Pipeline Parallelism/Expected Performance.md @@ -13,7 +13,7 @@ timestamp: 2026-08-14 This page estimates the speed of a 25-40B model split across two machines over ethernet. It uses the bandwidth math in [[Ethernet Bandwidth Math]] -and the mixed-vendor measurements in [[Multi-Node Runtimes]]. +and the mixed-vendor measurements in [[x4-pair-experiment]]. ## The Bandwidth Ceiling Is Not the Limit @@ -93,5 +93,5 @@ roughly 200k-260k tokens across both nodes. ## See Also - [[Ethernet Bandwidth Math]] -- [[Multi-Node Runtimes]] +- [[Qwen3.8 PP Deployment]] - [[Sample Script Set]] diff --git a/serverAppConfigs-docs/LLM/Pipeline Parallelism/Multi-Node Runtimes.md b/serverAppConfigs-docs/LLM/Pipeline Parallelism/Multi-Node Runtimes.md deleted file mode 100644 index b99a4e6..0000000 --- a/serverAppConfigs-docs/LLM/Pipeline Parallelism/Multi-Node Runtimes.md +++ /dev/null @@ -1,257 +0,0 @@ ---- -type: reference -title: Multi-Node Runtimes -description: "How each two-machine runtime is built and launched, what it can combine with, and the mixed-vendor verdicts." -tags: - - pipeline-parallelism - - llamacpp - - vllm - - speculative -timestamp: 2026-10-19 ---- - -# Multi-Node Runtimes - -Two runtimes can put one model on two machines: llama.cpp over the ggml-rpc -backend, and vLLM with pipeline parallelism. This page owns the cross-runtime -facts — how each is built and launched, what each can and cannot be combined -with, and the mixed-vendor verdicts. Facts owned by a sibling page are linked, -not restated. Bytes-per-token arithmetic and link budgets belong to -[[Ethernet Bandwidth Math]], throughput estimates to [[Expected Performance]], -the real deployment and its verified gotchas to [[Qwen3.8 PP Deployment]], the -educational scripts to [[Sample Script Set]], drafter mechanisms to -[[DFlash DSpark Speculative Decoding]], single-host multi-GPU placement to -[[Llama Cpp Multi-Device Splits]]. - -## What each runtime is - -Both distribute by **layers**, not inside layers. - -| | llama.cpp over ggml-rpc | vLLM pipeline parallelism | -| --- | --- | --- | -| unit of distribution | a contiguous slice of layers (`--split-mode layer`, the default) | a contiguous slice of layers (`--pipeline-parallel-size`) | -| process model | one `llama-server`/`llama-cli` on the head; a `ggml-rpc-server` per worker | one `vllm serve` per node, one engine across them | -| what crosses the wire | whatever graph work the head offloads to the remote devices | the hidden state at each stage boundary, point to point | -| KV cache | distributed with the weights; the KV of a layer lives with the owner of that layer | each stage holds the KV of the layers it owns | -| speculation together with the split | yes in llama.cpp (see below) | no, per vLLM's own docs | - -Upstream's own summary is why layer-wise is the ethernet choice: pipeline -parallel "minimizes data transfers between GPUs but requires many tokens to -scale well", while tensor parallel "does multiple cross-GPU reductions per -layer … much more bottlenecked by the GPU interconnect speed". - -Do not carry the tensor path across machines: `--split-mode tensor` is -experimental, needs flash attention, rejects quantized KV, and is not -implemented for the MoE and hybrid architectures the guide lists — and -per-tensor placement across vendors is not expressible (§1 of [[Llama Cpp Multi-Device Splits]]). - -## llama.cpp over ggml-rpc - -### Build and launch - -Both sides take `-DGGML_RPC=ON` next to that host's own backend: - -```bash -cmake .. -DGGML_CUDA=ON -DGGML_RPC=ON # a CUDA worker; -DGGML_HIP=ON for an AMD one -cmake --build . --config Release -``` - -Worker side — `ggml-rpc-server` exposes ggml devices on the remote host: - -```bash -bin/ggml-rpc-server # endpoint 127.0.0.1:50052 by default -bin/ggml-rpc-server --device CUDA0 -c # or CUDA_VISIBLE_DEVICES=0; -c = local cache -``` - -- It exposes every accelerator it can see; a host with no accelerator exposes a - single `CPU` device. `--device` and `CUDA_VISIBLE_DEVICES` are equivalent - ways to narrow that set. -- `-c` keeps large tensors in `$HOME/.cache/llama.cpp/rpc` (redirect with - `LLAMA_CACHE`) so a big model does not re-stream per load; that cache - filling the worker disk is a recorded fleet incident ([[Qwen3.8 PP Deployment]]). - -Head side — build the local backends plus `-DGGML_RPC=ON`, then name the -workers: - -```bash -llama-server -m model.gguf --rpc 192.168.88.10:50052,192.168.88.11:50052 -ngl 99 -``` - -`--rpc` is a comma-separated list of `host:port`, one entry per -`ggml-rpc-server`; the head's own devices stay in the pool next to the remote -ones. Launch scripts for this shape: [[Sample Script Set]]. - -### Split, context, KV - -- Default placement: weights **and** KV across all devices, local and remote, - "in proportion to each device's available memory". Override with - `--tensor-split`, whose proportions follow the order in `--device`. Why the - fleet always pins it (a Strix reports a huge GTT window, so the automatic - split lands on the slow device) is [[Qwen3.8 PP Deployment]]'s. -- `-sm layer` (the default) is pipeline parallelism: each device holds a - contiguous slice of layers and the KV cache of a layer lives on the device - that owns that layer. -- KV types are the usual `-ctk` / `-ctv`, `f16` by default. `--fit` is on by - default and sizes unset arguments to device memory; turn it off to hold a - context you set on purpose. Context is one global `--ctx-size` — the RPC - README documents no per-node context or per-node KV knob. -- Upstream's out-of-memory ladder (stated for tensor mode, useful order - anyway): lower `--ctx-size`, then `--parallel`, then `--n-gpu-layers`. -- Transport is TCP; RDMA is negotiated at the handshake with no flag changes - and falls back to TCP unless both peers can use it (Linux RoCEv2 via - `libibverbs`, macOS RDMA over Thunderbolt 5 via `librdma`). It is point to - point — name the peer's RDMA-capable address in `--rpc` or the session stays - on TCP. `GGML_RPC_NO_RDMA=1` forces TCP; `GGML_RPC_DEBUG` traces the backend. - -Upstream calls the backend a proof of concept, "fragile and insecure" (see the -Security section). Two limits shape the design: decode is bound by the per-step -round trip, not by the link's bytes ([[Ethernet Bandwidth Math]]), and every -node must run the identical build or the handshake hangs ([[Qwen3.8 PP Deployment]]). - -## vLLM pipeline parallelism - -> **Provenance.** Flags and mechanics below were recorded 2026-08-14 and are -> not re-checked upstream in this pass (see "Unverified in this pass"); the -> incompatibility lines, the method catalog and the issue history are re-verified. - -- `--pipeline-parallel-size` divides the layers into contiguous stages; - `--tensor-parallel-size` still works inside a node, so the canonical two-node - shape is tp = GPUs per node, pp = nodes. -- Two backends: Ray (`--distributed-executor-backend ray`, the worker joining - a `ray start --head`) or multiprocessing (`--distributed-executor-backend mp` - with `--nnodes`, `--node-rank`, `--master-addr`, `--headless` on the - follower). `mp` avoids the Ray dependency; Ray brings the cluster manager. -- What the interconnect must carry: one activation tensor per stage boundary - per forward pass, point to point — not a per-layer reduction. That is the - whole cross-node traffic story; the byte arithmetic and link budgets are - [[Ethernet Bandwidth Math]]. Pin the NIC vLLM talks on - (`NCCL_SOCKET_IFNAME`) rather than letting it guess an interface. -- The PP bubble: with N stages and one sequence in flight, one stage works - while N-1 idle, so pp=2 costs half the machine at batch one. Batching is the - cure — pipeline parallelism "requires many tokens to scale well" — and the - microbatch formula is [[Ethernet Bandwidth Math]] §5.3. -- KV and context: each stage keeps the KV of its own layers, so the pool grows - with the stage count, and `--max-model-len` is not reduced by the split - (recorded 2026-08-14, unverified here). -- What it cannot combine with, in vLLM's words: "Pipeline parallelism is not - composable with speculative decoding as of `vllm<=0.15.0`", and "Speculative - decoding with a draft models is not supported in `vllm<=0.10.0`". Composing - the two has been requested since 2024 and closed stale (#6911, #10615); PP - also carries a Qwen3.5 bug report closed stale without a fix (#36643). -- Method catalog at v0.25.1: EAGLE, MTP, draft model, PARD, MLP, n-gram, suffix - decoding, hidden-state extraction, a custom proposer class, dynamic - speculative decoding; `dflash` is an accepted `method` value. - -## Speculative decoding across the split - -llama.cpp chains several implementations behind one comma-separated -`--spec-type` list; the split does not change the chain, only where its passes -run. Mechanisms, chaining laws and drafter KV sizing are -[[DFlash DSpark Speculative Decoding]]'s; only the split interaction is here. - -| Method | llama.cpp type | Over ggml-rpc | With vLLM PP | Why | -| --- | --- | --- | --- | --- | -| MTP | `draft-mtp` | works; not documented upstream, but run this way on this fleet's split head with acceptance measured ([[Qwen3.8 PP Deployment]]) | no | Context-backed drafter: it needs a draft context and runs in the head process, so RPC devices are simply more devices under it | -| DFlash | `draft-dflash` | same gate, same status as MTP | no | Same draft-context gate (`ctx_dft`); the drafter's KV lands on whichever device the draft context is built on — the one thing to check on a split | -| ngram-mod | `ngram-mod` | safest of the four: no draft model, no draft context, no remote draft pass | no | Draftless — proposals come from cached token n-grams in the head, so the split adds no device work | -| DSpark | `draft-dspark` | **unproven over RPC** — not a missing feature | no | `draft-dspark` exists upstream; nobody has measured it with the target split across two machines | - -**Correction (2026-10-19).** A retired stub in this directory asserted "DSpark -is vLLM-only." False. `draft-dspark` is in upstream llama.cpp — parsed in -`common/arg.cpp`, mapped to `COMMON_SPECULATIVE_TYPE_DRAFT_DSPARK` in -`common/speculative.cpp`, where it carries a Markov head and an anchor-first -block layout and aborts with `DSpark draft has no confidence head: please set ---spec-draft-p-min 0` when the drafter has no confidence head. What is unproven -is running it over RPC, and that is the honest limit of the claim. - -The vLLM column is uniform because that statement is blanket: it names pipeline -parallelism, not a method, so n-gram and suffix are excluded as written. vLLM -gives no mechanism with it; "the drafter must sit next to the target's layers, -which a stage split separates" is a reading, not upstream's words. - -Upstream likewise documents neither for nor against `--rpc` plus -`--spec-type`. What the implementation does say is structural: every -speculative implementation is built in the head process against the head's -device set, and draftless types need no draft context at all. Routing draft -passes to a remote worker, however, has never been tested. - -## Mixed-vendor reality - -What works today: - -- **One vendor per node.** RPC is backend-agnostic by construction: upstream's - own diagram has a host exposing CUDA0+CUDA1, a host exposing Metal, and a - host exposing CUDA0+CPU, all driven by one main host over TCP. Each node - builds its own backend plus `-DGGML_RPC=ON`; nothing speaks CUDA-to-ROCm. -- **Identical build tag on every node**, worker restarted before head. The - verified version-skew gotcha is [[Qwen3.8 PP Deployment]]'s. -- **Speculation on the head** while the split carries the target (table above). -- **RDMA when both peers have it**, auto-negotiated at the handshake; the size - of the gain is third-party and unverified here. - -Unproven or broken: - -- **One binary holding both CUDA and ROCm for a same-host layer split**: - reported to compile and then list only CUDA devices (issue #11506, - third-party, unverified). Vulkan is the reported single-binary cross-vendor - path, and single-host placement is [[Llama Cpp Multi-Device Splits]]'s, whose - verdict is that per-tensor placement across a vendor mix never works. -- **vLLM across a CUDA node and a ROCm node**: no known experiment; both ends - run the same backend (recorded 2026-08-14). AMD's own two-Halo clustering - guide is tensor parallelism over RCCL, not a mixed-vendor split. -- **DSpark over RPC**: unproven, see the correction above. -- **The exact NVIDIA-desktop + Strix-Halo-NUC pair has no published - benchmark**, and the third-party mixed-vendor figures it is estimated from - (Metal+CUDA over 10 GbE, two Strix Halos over USB4, a three-card CUDA+ROCm set - over 1 GbE, a Vulkan triple in one box, an AMD four-node Strix cluster) live on - [[Expected Performance]] and [[Ethernet Bandwidth Math]], each named with its - model and hardware. None was re-verified in this pass. - -## Security - -`ggml-rpc-server` has **no authentication and no encryption**. Upstream's own -warning: the backend is "in a proof-of-concept development stage. As such, the -functionality is fragile and insecure. Never run the RPC server on an open -network or in a sensitive environment!" It binds `127.0.0.1:50052` by default — -safe by accident. Bind a routable address and any host that can reach the port -can allocate memory there and supply the tensors it computes on. Trusted LAN -only, firewalled or tunnelled; never a public address. - -## Unverified in this pass - -Every item was first recorded 2026-08-14, in the research that preceded this -page. Nothing is silently deleted; none is backed by the sources cited below. - -- **RPC protocol version and handshake rule** — recorded as protocol `5.0.0` - with "major must match, server minor must be ≥ client". The README's own - banner prints `Starting RPC server v3.0.0`; the conflict is unresolved. -- **RPC internals** — a hard cap of 16 servers, synchronous graph compute, no - tensor copy between two RPC servers, workers loading tensors on first use, a - hash-keyed cache above roughly 10 MB, asserts on quantized tensors whose rows - are not 512-aligned, and `-rpc` on `llama-bench` (the README names only - `llama-cli` and `llama-server`). -- **vLLM flag surface** — `--pipeline-parallel-size`, `--tensor-parallel-size`, - `--distributed-executor-backend {ray,mp}`, `--nnodes`, `--node-rank`, - `--master-addr`, `--headless`, the `ray start --head` join, - `NCCL_SOCKET_IFNAME`. -- **vLLM mechanics** — one activation handoff per stage boundary per forward - pass, a worked ~512 MB handoff for a hidden-5120 model at batch 8 and 2048 - positions, `--max-model-len` unchanged by PP, ~50% idle at pp=2 batch one. -- **vLLM throughput verdicts** for a 27B across two nodes over gigabit-class - links; superseded as an estimate by [[Expected Performance]]. -- **Mixed-vendor claims** — a CUDA+HIP build listing only CUDA devices - (#11506), Vulkan as the universal cross-vendor backend, AMD's two-Halo - clustering guide being RCCL tensor parallelism, WiFi costing about ten times - wired, the slowest hop setting the decode pace, RDMA worth a few percent. -- **Draft over RPC** — that the draft model stays on the head, and that routing - draft passes to a remote worker has never been confirmed. - -## Citations - -- [llama.cpp RPC README](https://github.com/ggml-org/llama.cpp/blob/master/tools/rpc/README.md), accessed 2026-10-19 — server usage and device exposure, `-DGGML_RPC=ON` on both sides, the `--rpc host:port` list, proportional weight+KV split and `--tensor-split`, the `-c` local cache, RDMA negotiation and `GGML_RPC_NO_RDMA`, the proof-of-concept and no-authentication warning. -- [llama.cpp multi-GPU guide](https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md), accessed 2026-10-19 — split modes (`-sm layer` = pipeline parallelism, KV with the layer owner), `-ts`/`-ngl`/`-ctk`/`-ctv`/`-fit`, the pipeline-versus-tensor summary, the tensor-mode limits. -- [vLLM speculative decoding docs, v0.25.1](https://docs.vllm.ai/en/v0.25.1/features/speculative_decoding/), accessed 2026-10-19 — the pipeline-parallelism incompatibility statement, the draft-model version floor, the method catalog. -- [llama.cpp `common/speculative.cpp`](https://github.com/ggml-org/llama.cpp/blob/master/common/speculative.cpp), accessed 2026-10-19 — the speculative implementation set including `draft-dspark`, its draft-context gate, its confidence-head error. -- vLLM issue #6911, "[Feature]: Combine pipeline parallelism with speculative decoding", opened 2024-07-29, closed not-planned 2025-03-28, accessed 2026-10-19. -- vLLM issue #10615, "[Usage]: Does speculative decoding support pipeline parallelism?", opened 2024-11-25, closed not-planned 2025-04-18, accessed 2026-10-19. -- vLLM issue #36643, "[Bug]: Qwen3.5 does not work with pipeline parallelism", opened 2026-03-10, closed not-planned 2026-09-24, accessed 2026-10-19. diff --git a/serverAppConfigs-docs/LLM/Pipeline Parallelism/Sample Script Set.md b/serverAppConfigs-docs/LLM/Pipeline Parallelism/Sample Script Set.md index 24313f6..37a1377 100644 --- a/serverAppConfigs-docs/LLM/Pipeline Parallelism/Sample Script Set.md +++ b/serverAppConfigs-docs/LLM/Pipeline Parallelism/Sample Script Set.md @@ -255,7 +255,7 @@ is 30 GiB. ## See Also -- [[Multi-Node Runtimes]] +- [[Qwen3.8 PP Deployment]] - [[Expected Performance]] - [[Ethernet Bandwidth Math]] - [[DFlash DSpark Speculative Decoding]] \ No newline at end of file