--- type: reference title: Multi-Node Runtimes description: "How each two-machine runtime is built and launched, what it can combine with, and the mixed-vendor verdicts." tags: - pipeline-parallelism - llamacpp - vllm - speculative timestamp: 2026-10-19 --- # Multi-Node Runtimes Two runtimes can put one model on two machines: llama.cpp over the ggml-rpc backend, and vLLM with pipeline parallelism. This page owns the cross-runtime facts — how each is built and launched, what each can and cannot be combined with, and the mixed-vendor verdicts. Facts owned by a sibling page are linked, not restated. Bytes-per-token arithmetic and link budgets belong to [[Ethernet Bandwidth Math]], throughput estimates to [[Expected Performance]], the real deployment and its verified gotchas to [[Qwen3.8 PP Deployment]], the educational scripts to [[Sample Script Set]], drafter mechanisms to [[DFlash DSpark Speculative Decoding]], single-host multi-GPU placement to [[Llama Cpp Multi-Device Splits]]. ## What each runtime is Both distribute by **layers**, not inside layers. | | llama.cpp over ggml-rpc | vLLM pipeline parallelism | | --- | --- | --- | | unit of distribution | a contiguous slice of layers (`--split-mode layer`, the default) | a contiguous slice of layers (`--pipeline-parallel-size`) | | process model | one `llama-server`/`llama-cli` on the head; a `ggml-rpc-server` per worker | one `vllm serve` per node, one engine across them | | what crosses the wire | whatever graph work the head offloads to the remote devices | the hidden state at each stage boundary, point to point | | KV cache | distributed with the weights; the KV of a layer lives with the owner of that layer | each stage holds the KV of the layers it owns | | speculation together with the split | yes in llama.cpp (see below) | no, per vLLM's own docs | Upstream's own summary is why layer-wise is the ethernet choice: pipeline parallel "minimizes data transfers between GPUs but requires many tokens to scale well", while tensor parallel "does multiple cross-GPU reductions per layer … much more bottlenecked by the GPU interconnect speed". Do not carry the tensor path across machines: `--split-mode tensor` is experimental, needs flash attention, rejects quantized KV, and is not implemented for the MoE and hybrid architectures the guide lists — and per-tensor placement across vendors is not expressible (§1 of [[Llama Cpp Multi-Device Splits]]). ## llama.cpp over ggml-rpc ### Build and launch Both sides take `-DGGML_RPC=ON` next to that host's own backend: ```bash cmake .. -DGGML_CUDA=ON -DGGML_RPC=ON # a CUDA worker; -DGGML_HIP=ON for an AMD one cmake --build . --config Release ``` Worker side — `ggml-rpc-server` exposes ggml devices on the remote host: ```bash bin/ggml-rpc-server # endpoint 127.0.0.1:50052 by default bin/ggml-rpc-server --device CUDA0 -c # or CUDA_VISIBLE_DEVICES=0; -c = local cache ``` - It exposes every accelerator it can see; a host with no accelerator exposes a single `CPU` device. `--device` and `CUDA_VISIBLE_DEVICES` are equivalent ways to narrow that set. - `-c` keeps large tensors in `$HOME/.cache/llama.cpp/rpc` (redirect with `LLAMA_CACHE`) so a big model does not re-stream per load; that cache filling the worker disk is a recorded fleet incident ([[Qwen3.8 PP Deployment]]). Head side — build the local backends plus `-DGGML_RPC=ON`, then name the workers: ```bash llama-server -m model.gguf --rpc 192.168.88.10:50052,192.168.88.11:50052 -ngl 99 ``` `--rpc` is a comma-separated list of `host:port`, one entry per `ggml-rpc-server`; the head's own devices stay in the pool next to the remote ones. Launch scripts for this shape: [[Sample Script Set]]. ### Split, context, KV - Default placement: weights **and** KV across all devices, local and remote, "in proportion to each device's available memory". Override with `--tensor-split`, whose proportions follow the order in `--device`. Why the fleet always pins it (a Strix reports a huge GTT window, so the automatic split lands on the slow device) is [[Qwen3.8 PP Deployment]]'s. - `-sm layer` (the default) is pipeline parallelism: each device holds a contiguous slice of layers and the KV cache of a layer lives on the device that owns that layer. - KV types are the usual `-ctk` / `-ctv`, `f16` by default. `--fit` is on by default and sizes unset arguments to device memory; turn it off to hold a context you set on purpose. Context is one global `--ctx-size` — the RPC README documents no per-node context or per-node KV knob. - Upstream's out-of-memory ladder (stated for tensor mode, useful order anyway): lower `--ctx-size`, then `--parallel`, then `--n-gpu-layers`. - Transport is TCP; RDMA is negotiated at the handshake with no flag changes and falls back to TCP unless both peers can use it (Linux RoCEv2 via `libibverbs`, macOS RDMA over Thunderbolt 5 via `librdma`). It is point to point — name the peer's RDMA-capable address in `--rpc` or the session stays on TCP. `GGML_RPC_NO_RDMA=1` forces TCP; `GGML_RPC_DEBUG` traces the backend. Upstream calls the backend a proof of concept, "fragile and insecure" (see the Security section). Two limits shape the design: decode is bound by the per-step round trip, not by the link's bytes ([[Ethernet Bandwidth Math]]), and every node must run the identical build or the handshake hangs ([[Qwen3.8 PP Deployment]]). ## vLLM pipeline parallelism > **Provenance.** Flags and mechanics below were recorded 2026-08-14 and are > not re-checked upstream in this pass (see "Unverified in this pass"); the > incompatibility lines, the method catalog and the issue history are re-verified. - `--pipeline-parallel-size` divides the layers into contiguous stages; `--tensor-parallel-size` still works inside a node, so the canonical two-node shape is tp = GPUs per node, pp = nodes. - Two backends: Ray (`--distributed-executor-backend ray`, the worker joining a `ray start --head`) or multiprocessing (`--distributed-executor-backend mp` with `--nnodes`, `--node-rank`, `--master-addr`, `--headless` on the follower). `mp` avoids the Ray dependency; Ray brings the cluster manager. - What the interconnect must carry: one activation tensor per stage boundary per forward pass, point to point — not a per-layer reduction. That is the whole cross-node traffic story; the byte arithmetic and link budgets are [[Ethernet Bandwidth Math]]. Pin the NIC vLLM talks on (`NCCL_SOCKET_IFNAME`) rather than letting it guess an interface. - The PP bubble: with N stages and one sequence in flight, one stage works while N-1 idle, so pp=2 costs half the machine at batch one. Batching is the cure — pipeline parallelism "requires many tokens to scale well" — and the microbatch formula is [[Ethernet Bandwidth Math]] §5.3. - KV and context: each stage keeps the KV of its own layers, so the pool grows with the stage count, and `--max-model-len` is not reduced by the split (recorded 2026-08-14, unverified here). - What it cannot combine with, in vLLM's words: "Pipeline parallelism is not composable with speculative decoding as of `vllm<=0.15.0`", and "Speculative decoding with a draft models is not supported in `vllm<=0.10.0`". Composing the two has been requested since 2024 and closed stale (#6911, #10615); PP also carries a Qwen3.5 bug report closed stale without a fix (#36643). - Method catalog at v0.25.1: EAGLE, MTP, draft model, PARD, MLP, n-gram, suffix decoding, hidden-state extraction, a custom proposer class, dynamic speculative decoding; `dflash` is an accepted `method` value. ## Speculative decoding across the split llama.cpp chains several implementations behind one comma-separated `--spec-type` list; the split does not change the chain, only where its passes run. Mechanisms, chaining laws and drafter KV sizing are [[DFlash DSpark Speculative Decoding]]'s; only the split interaction is here. | Method | llama.cpp type | Over ggml-rpc | With vLLM PP | Why | | --- | --- | --- | --- | --- | | MTP | `draft-mtp` | works; not documented upstream, but run this way on this fleet's split head with acceptance measured ([[Qwen3.8 PP Deployment]]) | no | Context-backed drafter: it needs a draft context and runs in the head process, so RPC devices are simply more devices under it | | DFlash | `draft-dflash` | same gate, same status as MTP | no | Same draft-context gate (`ctx_dft`); the drafter's KV lands on whichever device the draft context is built on — the one thing to check on a split | | ngram-mod | `ngram-mod` | safest of the four: no draft model, no draft context, no remote draft pass | no | Draftless — proposals come from cached token n-grams in the head, so the split adds no device work | | DSpark | `draft-dspark` | **unproven over RPC** — not a missing feature | no | `draft-dspark` exists upstream; nobody has measured it with the target split across two machines | **Correction (2026-10-19).** A retired stub in this directory asserted "DSpark is vLLM-only." False. `draft-dspark` is in upstream llama.cpp — parsed in `common/arg.cpp`, mapped to `COMMON_SPECULATIVE_TYPE_DRAFT_DSPARK` in `common/speculative.cpp`, where it carries a Markov head and an anchor-first block layout and aborts with `DSpark draft has no confidence head: please set --spec-draft-p-min 0` when the drafter has no confidence head. What is unproven is running it over RPC, and that is the honest limit of the claim. The vLLM column is uniform because that statement is blanket: it names pipeline parallelism, not a method, so n-gram and suffix are excluded as written. vLLM gives no mechanism with it; "the drafter must sit next to the target's layers, which a stage split separates" is a reading, not upstream's words. Upstream likewise documents neither for nor against `--rpc` plus `--spec-type`. What the implementation does say is structural: every speculative implementation is built in the head process against the head's device set, and draftless types need no draft context at all. Routing draft passes to a remote worker, however, has never been tested. ## Mixed-vendor reality What works today: - **One vendor per node.** RPC is backend-agnostic by construction: upstream's own diagram has a host exposing CUDA0+CUDA1, a host exposing Metal, and a host exposing CUDA0+CPU, all driven by one main host over TCP. Each node builds its own backend plus `-DGGML_RPC=ON`; nothing speaks CUDA-to-ROCm. - **Identical build tag on every node**, worker restarted before head. The verified version-skew gotcha is [[Qwen3.8 PP Deployment]]'s. - **Speculation on the head** while the split carries the target (table above). - **RDMA when both peers have it**, auto-negotiated at the handshake; the size of the gain is third-party and unverified here. Unproven or broken: - **One binary holding both CUDA and ROCm for a same-host layer split**: reported to compile and then list only CUDA devices (issue #11506, third-party, unverified). Vulkan is the reported single-binary cross-vendor path, and single-host placement is [[Llama Cpp Multi-Device Splits]]'s, whose verdict is that per-tensor placement across a vendor mix never works. - **vLLM across a CUDA node and a ROCm node**: no known experiment; both ends run the same backend (recorded 2026-08-14). AMD's own two-Halo clustering guide is tensor parallelism over RCCL, not a mixed-vendor split. - **DSpark over RPC**: unproven, see the correction above. - **The exact NVIDIA-desktop + Strix-Halo-NUC pair has no published benchmark**, and the third-party mixed-vendor figures it is estimated from (Metal+CUDA over 10 GbE, two Strix Halos over USB4, a three-card CUDA+ROCm set over 1 GbE, a Vulkan triple in one box, an AMD four-node Strix cluster) live on [[Expected Performance]] and [[Ethernet Bandwidth Math]], each named with its model and hardware. None was re-verified in this pass. ## Security `ggml-rpc-server` has **no authentication and no encryption**. Upstream's own warning: the backend is "in a proof-of-concept development stage. As such, the functionality is fragile and insecure. Never run the RPC server on an open network or in a sensitive environment!" It binds `127.0.0.1:50052` by default — safe by accident. Bind a routable address and any host that can reach the port can allocate memory there and supply the tensors it computes on. Trusted LAN only, firewalled or tunnelled; never a public address. ## Unverified in this pass Every item was first recorded 2026-08-14, in the research that preceded this page. Nothing is silently deleted; none is backed by the sources cited below. - **RPC protocol version and handshake rule** — recorded as protocol `5.0.0` with "major must match, server minor must be ≥ client". The README's own banner prints `Starting RPC server v3.0.0`; the conflict is unresolved. - **RPC internals** — a hard cap of 16 servers, synchronous graph compute, no tensor copy between two RPC servers, workers loading tensors on first use, a hash-keyed cache above roughly 10 MB, asserts on quantized tensors whose rows are not 512-aligned, and `-rpc` on `llama-bench` (the README names only `llama-cli` and `llama-server`). - **vLLM flag surface** — `--pipeline-parallel-size`, `--tensor-parallel-size`, `--distributed-executor-backend {ray,mp}`, `--nnodes`, `--node-rank`, `--master-addr`, `--headless`, the `ray start --head` join, `NCCL_SOCKET_IFNAME`. - **vLLM mechanics** — one activation handoff per stage boundary per forward pass, a worked ~512 MB handoff for a hidden-5120 model at batch 8 and 2048 positions, `--max-model-len` unchanged by PP, ~50% idle at pp=2 batch one. - **vLLM throughput verdicts** for a 27B across two nodes over gigabit-class links; superseded as an estimate by [[Expected Performance]]. - **Mixed-vendor claims** — a CUDA+HIP build listing only CUDA devices (#11506), Vulkan as the universal cross-vendor backend, AMD's two-Halo clustering guide being RCCL tensor parallelism, WiFi costing about ten times wired, the slowest hop setting the decode pace, RDMA worth a few percent. - **Draft over RPC** — that the draft model stays on the head, and that routing draft passes to a remote worker has never been confirmed. ## Citations - [llama.cpp RPC README](https://github.com/ggml-org/llama.cpp/blob/master/tools/rpc/README.md), accessed 2026-10-19 — server usage and device exposure, `-DGGML_RPC=ON` on both sides, the `--rpc host:port` list, proportional weight+KV split and `--tensor-split`, the `-c` local cache, RDMA negotiation and `GGML_RPC_NO_RDMA`, the proof-of-concept and no-authentication warning. - [llama.cpp multi-GPU guide](https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md), accessed 2026-10-19 — split modes (`-sm layer` = pipeline parallelism, KV with the layer owner), `-ts`/`-ngl`/`-ctk`/`-ctv`/`-fit`, the pipeline-versus-tensor summary, the tensor-mode limits. - [vLLM speculative decoding docs, v0.25.1](https://docs.vllm.ai/en/v0.25.1/features/speculative_decoding/), accessed 2026-10-19 — the pipeline-parallelism incompatibility statement, the draft-model version floor, the method catalog. - [llama.cpp `common/speculative.cpp`](https://github.com/ggml-org/llama.cpp/blob/master/common/speculative.cpp), accessed 2026-10-19 — the speculative implementation set including `draft-dspark`, its draft-context gate, its confidence-head error. - vLLM issue #6911, "[Feature]: Combine pipeline parallelism with speculative decoding", opened 2024-07-29, closed not-planned 2025-03-28, accessed 2026-10-19. - vLLM issue #10615, "[Usage]: Does speculative decoding support pipeline parallelism?", opened 2024-11-25, closed not-planned 2025-04-18, accessed 2026-10-19. - vLLM issue #36643, "[Bug]: Qwen3.5 does not work with pipeline parallelism", opened 2026-03-10, closed not-planned 2026-09-24, accessed 2026-10-19.