serverAppConfigs/serverAppConfigs-docs/LLM/Pipeline Parallelism/Multi-Node Runtimes.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	Multi-Node Runtimes	How each two-machine runtime is built and launched, what it can combine with, and the mixed-vendor verdicts.	pipeline-parallelism llamacpp vllm speculative	2026-10-19

Multi-Node Runtimes

Two runtimes can put one model on two machines: llama.cpp over the ggml-rpc backend, and vLLM with pipeline parallelism. This page owns the cross-runtime facts — how each is built and launched, what each can and cannot be combined with, and the mixed-vendor verdicts. Facts owned by a sibling page are linked, not restated. Bytes-per-token arithmetic and link budgets belong to Ethernet Bandwidth Math, throughput estimates to Expected Performance, the real deployment and its verified gotchas to Qwen3.8 PP Deployment, the educational scripts to Sample Script Set, drafter mechanisms to DFlash DSpark Speculative Decoding, single-host multi-GPU placement to Llama Cpp Multi-Device Splits.

What each runtime is

Both distribute by layers, not inside layers.

	llama.cpp over ggml-rpc	vLLM pipeline parallelism
unit of distribution	a contiguous slice of layers (--split-mode layer, the default)	a contiguous slice of layers (--pipeline-parallel-size)
process model	one llama-server/llama-cli on the head; a ggml-rpc-server per worker	one vllm serve per node, one engine across them
what crosses the wire	whatever graph work the head offloads to the remote devices	the hidden state at each stage boundary, point to point
KV cache	distributed with the weights; the KV of a layer lives with the owner of that layer	each stage holds the KV of the layers it owns
speculation together with the split	yes in llama.cpp (see below)	no, per vLLM's own docs

Upstream's own summary is why layer-wise is the ethernet choice: pipeline parallel "minimizes data transfers between GPUs but requires many tokens to scale well", while tensor parallel "does multiple cross-GPU reductions per layer … much more bottlenecked by the GPU interconnect speed".

Do not carry the tensor path across machines: --split-mode tensor is experimental, needs flash attention, rejects quantized KV, and is not implemented for the MoE and hybrid architectures the guide lists — and per-tensor placement across vendors is not expressible (§1 of Llama Cpp Multi-Device Splits).

llama.cpp over ggml-rpc

Build and launch

Both sides take -DGGML_RPC=ON next to that host's own backend:

cmake .. -DGGML_CUDA=ON -DGGML_RPC=ON    # a CUDA worker; -DGGML_HIP=ON for an AMD one
cmake --build . --config Release

Worker side — ggml-rpc-server exposes ggml devices on the remote host:

bin/ggml-rpc-server                      # endpoint 127.0.0.1:50052 by default
bin/ggml-rpc-server --device CUDA0 -c    # or CUDA_VISIBLE_DEVICES=0; -c = local cache

• It exposes every accelerator it can see; a host with no accelerator exposes a single CPU device. --device and CUDA_VISIBLE_DEVICES are equivalent ways to narrow that set.
• -c keeps large tensors in $HOME/.cache/llama.cpp/rpc (redirect with LLAMA_CACHE) so a big model does not re-stream per load; that cache filling the worker disk is a recorded fleet incident (Qwen3.8 PP Deployment).

Head side — build the local backends plus -DGGML_RPC=ON, then name the workers:

llama-server -m model.gguf --rpc 192.168.88.10:50052,192.168.88.11:50052 -ngl 99

--rpc is a comma-separated list of host:port, one entry per ggml-rpc-server; the head's own devices stay in the pool next to the remote ones. Launch scripts for this shape: Sample Script Set.

Split, context, KV

• Default placement: weights and KV across all devices, local and remote, "in proportion to each device's available memory". Override with --tensor-split, whose proportions follow the order in --device. Why the fleet always pins it (a Strix reports a huge GTT window, so the automatic split lands on the slow device) is Qwen3.8 PP Deployment's.
• -sm layer (the default) is pipeline parallelism: each device holds a contiguous slice of layers and the KV cache of a layer lives on the device that owns that layer.
• KV types are the usual -ctk / -ctv, f16 by default. --fit is on by default and sizes unset arguments to device memory; turn it off to hold a context you set on purpose. Context is one global --ctx-size — the RPC README documents no per-node context or per-node KV knob.
• Upstream's out-of-memory ladder (stated for tensor mode, useful order anyway): lower --ctx-size, then --parallel, then --n-gpu-layers.
• Transport is TCP; RDMA is negotiated at the handshake with no flag changes and falls back to TCP unless both peers can use it (Linux RoCEv2 via libibverbs, macOS RDMA over Thunderbolt 5 via librdma). It is point to point — name the peer's RDMA-capable address in --rpc or the session stays on TCP. GGML_RPC_NO_RDMA=1 forces TCP; GGML_RPC_DEBUG traces the backend.

Upstream calls the backend a proof of concept, "fragile and insecure" (see the Security section). Two limits shape the design: decode is bound by the per-step round trip, not by the link's bytes (Ethernet Bandwidth Math), and every node must run the identical build or the handshake hangs (Qwen3.8 PP Deployment).

vLLM pipeline parallelism

Provenance. Flags and mechanics below were recorded 2026-08-14 and are not re-checked upstream in this pass (see "Unverified in this pass"); the incompatibility lines, the method catalog and the issue history are re-verified.

• --pipeline-parallel-size divides the layers into contiguous stages; --tensor-parallel-size still works inside a node, so the canonical two-node shape is tp = GPUs per node, pp = nodes.
• Two backends: Ray (--distributed-executor-backend ray, the worker joining a ray start --head) or multiprocessing (--distributed-executor-backend mp with --nnodes, --node-rank, --master-addr, --headless on the follower). mp avoids the Ray dependency; Ray brings the cluster manager.
• What the interconnect must carry: one activation tensor per stage boundary per forward pass, point to point — not a per-layer reduction. That is the whole cross-node traffic story; the byte arithmetic and link budgets are Ethernet Bandwidth Math. Pin the NIC vLLM talks on (NCCL_SOCKET_IFNAME) rather than letting it guess an interface.
• The PP bubble: with N stages and one sequence in flight, one stage works while N-1 idle, so pp=2 costs half the machine at batch one. Batching is the cure — pipeline parallelism "requires many tokens to scale well" — and the microbatch formula is Ethernet Bandwidth Math §5.3.
• KV and context: each stage keeps the KV of its own layers, so the pool grows with the stage count, and --max-model-len is not reduced by the split (recorded 2026-08-14, unverified here).
• What it cannot combine with, in vLLM's words: "Pipeline parallelism is not composable with speculative decoding as of vllm<=0.15.0", and "Speculative decoding with a draft models is not supported in vllm<=0.10.0". Composing the two has been requested since 2024 and closed stale (#6911, #10615); PP also carries a Qwen3.5 bug report closed stale without a fix (#36643).
• Method catalog at v0.25.1: EAGLE, MTP, draft model, PARD, MLP, n-gram, suffix decoding, hidden-state extraction, a custom proposer class, dynamic speculative decoding; dflash is an accepted method value.

Speculative decoding across the split

llama.cpp chains several implementations behind one comma-separated --spec-type list; the split does not change the chain, only where its passes run. Mechanisms, chaining laws and drafter KV sizing are DFlash DSpark Speculative Decoding's; only the split interaction is here.

Method	llama.cpp type	Over ggml-rpc	With vLLM PP	Why
MTP	draft-mtp	works; not documented upstream, but run this way on this fleet's split head with acceptance measured (Qwen3.8 PP Deployment)	no	Context-backed drafter: it needs a draft context and runs in the head process, so RPC devices are simply more devices under it
DFlash	draft-dflash	same gate, same status as MTP	no	Same draft-context gate (ctx_dft); the drafter's KV lands on whichever device the draft context is built on — the one thing to check on a split
ngram-mod	ngram-mod	safest of the four: no draft model, no draft context, no remote draft pass	no	Draftless — proposals come from cached token n-grams in the head, so the split adds no device work
DSpark	draft-dspark	unproven over RPC — not a missing feature	no	draft-dspark exists upstream; nobody has measured it with the target split across two machines

Correction (2026-10-19). A retired stub in this directory asserted "DSpark is vLLM-only." False. draft-dspark is in upstream llama.cpp — parsed in common/arg.cpp, mapped to COMMON_SPECULATIVE_TYPE_DRAFT_DSPARK in common/speculative.cpp, where it carries a Markov head and an anchor-first block layout and aborts with DSpark draft has no confidence head: please set --spec-draft-p-min 0 when the drafter has no confidence head. What is unproven is running it over RPC, and that is the honest limit of the claim.

The vLLM column is uniform because that statement is blanket: it names pipeline parallelism, not a method, so n-gram and suffix are excluded as written. vLLM gives no mechanism with it; "the drafter must sit next to the target's layers, which a stage split separates" is a reading, not upstream's words.

Upstream likewise documents neither for nor against --rpc plus --spec-type. What the implementation does say is structural: every speculative implementation is built in the head process against the head's device set, and draftless types need no draft context at all. Routing draft passes to a remote worker, however, has never been tested.

Mixed-vendor reality

What works today:

• One vendor per node. RPC is backend-agnostic by construction: upstream's own diagram has a host exposing CUDA0+CUDA1, a host exposing Metal, and a host exposing CUDA0+CPU, all driven by one main host over TCP. Each node builds its own backend plus -DGGML_RPC=ON; nothing speaks CUDA-to-ROCm.
• Identical build tag on every node, worker restarted before head. The verified version-skew gotcha is Qwen3.8 PP Deployment's.
• Speculation on the head while the split carries the target (table above).
• RDMA when both peers have it, auto-negotiated at the handshake; the size of the gain is third-party and unverified here.

Unproven or broken:

• One binary holding both CUDA and ROCm for a same-host layer split: reported to compile and then list only CUDA devices (issue #11506, third-party, unverified). Vulkan is the reported single-binary cross-vendor path, and single-host placement is Llama Cpp Multi-Device Splits's, whose verdict is that per-tensor placement across a vendor mix never works.
• vLLM across a CUDA node and a ROCm node: no known experiment; both ends run the same backend (recorded 2026-08-14). AMD's own two-Halo clustering guide is tensor parallelism over RCCL, not a mixed-vendor split.
• DSpark over RPC: unproven, see the correction above.
• The exact NVIDIA-desktop + Strix-Halo-NUC pair has no published benchmark, and the third-party mixed-vendor figures it is estimated from (Metal+CUDA over 10 GbE, two Strix Halos over USB4, a three-card CUDA+ROCm set over 1 GbE, a Vulkan triple in one box, an AMD four-node Strix cluster) live on Expected Performance and Ethernet Bandwidth Math, each named with its model and hardware. None was re-verified in this pass.

Security

ggml-rpc-server has no authentication and no encryption. Upstream's own warning: the backend is "in a proof-of-concept development stage. As such, the functionality is fragile and insecure. Never run the RPC server on an open network or in a sensitive environment!" It binds 127.0.0.1:50052 by default — safe by accident. Bind a routable address and any host that can reach the port can allocate memory there and supply the tensors it computes on. Trusted LAN only, firewalled or tunnelled; never a public address.

Unverified in this pass

Every item was first recorded 2026-08-14, in the research that preceded this page. Nothing is silently deleted; none is backed by the sources cited below.

• RPC protocol version and handshake rule — recorded as protocol 5.0.0 with "major must match, server minor must be ≥ client". The README's own banner prints Starting RPC server v3.0.0; the conflict is unresolved.
• RPC internals — a hard cap of 16 servers, synchronous graph compute, no tensor copy between two RPC servers, workers loading tensors on first use, a hash-keyed cache above roughly 10 MB, asserts on quantized tensors whose rows are not 512-aligned, and -rpc on llama-bench (the README names only llama-cli and llama-server).
• vLLM flag surface — --pipeline-parallel-size, --tensor-parallel-size, --distributed-executor-backend {ray,mp}, --nnodes, --node-rank, --master-addr, --headless, the ray start --head join, NCCL_SOCKET_IFNAME.
• vLLM mechanics — one activation handoff per stage boundary per forward pass, a worked ~512 MB handoff for a hidden-5120 model at batch 8 and 2048 positions, --max-model-len unchanged by PP, ~50% idle at pp=2 batch one.
• vLLM throughput verdicts for a 27B across two nodes over gigabit-class links; superseded as an estimate by Expected Performance.
• Mixed-vendor claims — a CUDA+HIP build listing only CUDA devices (#11506), Vulkan as the universal cross-vendor backend, AMD's two-Halo clustering guide being RCCL tensor parallelism, WiFi costing about ten times wired, the slowest hop setting the decode pace, RDMA worth a few percent.
• Draft over RPC — that the draft model stays on the head, and that routing draft passes to a remote worker has never been confirmed.

Citations

• llama.cpp RPC README, accessed 2026-10-19 — server usage and device exposure, -DGGML_RPC=ON on both sides, the --rpc host:port list, proportional weight+KV split and --tensor-split, the -c local cache, RDMA negotiation and GGML_RPC_NO_RDMA, the proof-of-concept and no-authentication warning.
• llama.cpp multi-GPU guide, accessed 2026-10-19 — split modes (-sm layer = pipeline parallelism, KV with the layer owner), -ts/-ngl/-ctk/-ctv/-fit, the pipeline-versus-tensor summary, the tensor-mode limits.
• vLLM speculative decoding docs, v0.25.1, accessed 2026-10-19 — the pipeline-parallelism incompatibility statement, the draft-model version floor, the method catalog.
• llama.cpp common/speculative.cpp, accessed 2026-10-19 — the speculative implementation set including draft-dspark, its draft-context gate, its confidence-head error.
• vLLM issue #6911, "[Feature]: Combine pipeline parallelism with speculative decoding", opened 2024-07-29, closed not-planned 2025-03-28, accessed 2026-10-19.
• vLLM issue #10615, "[Usage]: Does speculative decoding support pipeline parallelism?", opened 2024-11-25, closed not-planned 2025-04-18, accessed 2026-10-19.
• vLLM issue #36643, "[Bug]: Qwen3.5 does not work with pipeline parallelism", opened 2026-03-10, closed not-planned 2026-09-24, accessed 2026-10-19.