dank/serverAppConfigs
1
0
Fork 0
Code Issues 6 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(vault): documentation exists for point of use — destroy the ledgers (−13.7k lines) #31

Merged
dank merged 13 commits from docs/vault-destruction-20261019 into main 2026-10-19 10:59:51 -07:00
Conversation 0 Commits 13 Files Changed 157
+3 -260
3 changed files with 3 additions and 260 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Showing only changes of commit 2a12487653 - Show all commits
serverAppConfigs-docs/LLM/Pipeline Parallelism/Expected Performance.md
+2 -2
View File
@@ -13,7 +13,7 @@ timestamp: 2026-08-14
This page estimates the speed of a 25-40B model split across two machines
over ethernet. It uses the bandwidth math in [[Ethernet Bandwidth Math]]
and the mixed-vendor measurements in [[Multi-Node Runtimes]].
and the mixed-vendor measurements in [[x4-pair-experiment]].
## The Bandwidth Ceiling Is Not the Limit
@@ -93,5 +93,5 @@ roughly 200k-260k tokens across both nodes.
## See Also
- [[Ethernet Bandwidth Math]]
- [[Multi-Node Runtimes]]
- [[Qwen3.8 PP Deployment]]
- [[Sample Script Set]]
serverAppConfigs-docs/LLM/Pipeline Parallelism/Multi-Node Runtimes.md
-257
View File
@@ -1,257 +0,0 @@
---
type: reference
title: Multi-Node Runtimes
description: "How each two-machine runtime is built and launched, what it can combine with, and the mixed-vendor verdicts."
tags:
- pipeline-parallelism
- llamacpp
- vllm
- speculative
timestamp: 2026-10-19
---
# Multi-Node Runtimes
Two runtimes can put one model on two machines: llama.cpp over the ggml-rpc
backend, and vLLM with pipeline parallelism. This page owns the cross-runtime
facts — how each is built and launched, what each can and cannot be combined
with, and the mixed-vendor verdicts. Facts owned by a sibling page are linked,
not restated. Bytes-per-token arithmetic and link budgets belong to
[[Ethernet Bandwidth Math]], throughput estimates to [[Expected Performance]],
the real deployment and its verified gotchas to [[Qwen3.8 PP Deployment]], the
educational scripts to [[Sample Script Set]], drafter mechanisms to
[[DFlash DSpark Speculative Decoding]], single-host multi-GPU placement to
[[Llama Cpp Multi-Device Splits]].
## What each runtime is
Both distribute by **layers**, not inside layers.
| | llama.cpp over ggml-rpc | vLLM pipeline parallelism |
| --- | --- | --- |
| unit of distribution | a contiguous slice of layers (`--split-mode layer`, the default) | a contiguous slice of layers (`--pipeline-parallel-size`) |
| process model | one `llama-server`/`llama-cli` on the head; a `ggml-rpc-server` per worker | one `vllm serve` per node, one engine across them |
| what crosses the wire | whatever graph work the head offloads to the remote devices | the hidden state at each stage boundary, point to point |
| KV cache | distributed with the weights; the KV of a layer lives with the owner of that layer | each stage holds the KV of the layers it owns |
| speculation together with the split | yes in llama.cpp (see below) | no, per vLLM's own docs |
Upstream's own summary is why layer-wise is the ethernet choice: pipeline
parallel "minimizes data transfers between GPUs but requires many tokens to
scale well", while tensor parallel "does multiple cross-GPU reductions per
layer … much more bottlenecked by the GPU interconnect speed".
Do not carry the tensor path across machines: `--split-mode tensor` is
experimental, needs flash attention, rejects quantized KV, and is not
implemented for the MoE and hybrid architectures the guide lists — and
per-tensor placement across vendors is not expressible (§1 of [[Llama Cpp Multi-Device Splits]]).
## llama.cpp over ggml-rpc
### Build and launch
Both sides take `-DGGML_RPC=ON` next to that host's own backend:
```bash
cmake .. -DGGML_CUDA=ON -DGGML_RPC=ON # a CUDA worker; -DGGML_HIP=ON for an AMD one
cmake --build . --config Release
```
Worker side — `ggml-rpc-server` exposes ggml devices on the remote host:
```bash
bin/ggml-rpc-server # endpoint 127.0.0.1:50052 by default
bin/ggml-rpc-server --device CUDA0 -c # or CUDA_VISIBLE_DEVICES=0; -c = local cache
```
- It exposes every accelerator it can see; a host with no accelerator exposes a
single `CPU` device. `--device` and `CUDA_VISIBLE_DEVICES` are equivalent
ways to narrow that set.
- `-c` keeps large tensors in `$HOME/.cache/llama.cpp/rpc` (redirect with
`LLAMA_CACHE`) so a big model does not re-stream per load; that cache
filling the worker disk is a recorded fleet incident ([[Qwen3.8 PP Deployment]]).
Head side — build the local backends plus `-DGGML_RPC=ON`, then name the
workers:
```bash
llama-server -m model.gguf --rpc 192.168.88.10:50052,192.168.88.11:50052 -ngl 99
```
`--rpc` is a comma-separated list of `host:port`, one entry per
`ggml-rpc-server`; the head's own devices stay in the pool next to the remote
ones. Launch scripts for this shape: [[Sample Script Set]].
### Split, context, KV
- Default placement: weights **and** KV across all devices, local and remote,
"in proportion to each device's available memory". Override with
`--tensor-split`, whose proportions follow the order in `--device`. Why the
fleet always pins it (a Strix reports a huge GTT window, so the automatic
split lands on the slow device) is [[Qwen3.8 PP Deployment]]'s.
- `-sm layer` (the default) is pipeline parallelism: each device holds a
contiguous slice of layers and the KV cache of a layer lives on the device
that owns that layer.
- KV types are the usual `-ctk` / `-ctv`, `f16` by default. `--fit` is on by
default and sizes unset arguments to device memory; turn it off to hold a
context you set on purpose. Context is one global `--ctx-size` — the RPC
README documents no per-node context or per-node KV knob.
- Upstream's out-of-memory ladder (stated for tensor mode, useful order
anyway): lower `--ctx-size`, then `--parallel`, then `--n-gpu-layers`.
- Transport is TCP; RDMA is negotiated at the handshake with no flag changes
and falls back to TCP unless both peers can use it (Linux RoCEv2 via
`libibverbs`, macOS RDMA over Thunderbolt 5 via `librdma`). It is point to
point — name the peer's RDMA-capable address in `--rpc` or the session stays
on TCP. `GGML_RPC_NO_RDMA=1` forces TCP; `GGML_RPC_DEBUG` traces the backend.
Upstream calls the backend a proof of concept, "fragile and insecure" (see the
Security section). Two limits shape the design: decode is bound by the per-step
round trip, not by the link's bytes ([[Ethernet Bandwidth Math]]), and every
node must run the identical build or the handshake hangs ([[Qwen3.8 PP Deployment]]).
## vLLM pipeline parallelism
> **Provenance.** Flags and mechanics below were recorded 2026-08-14 and are
> not re-checked upstream in this pass (see "Unverified in this pass"); the
> incompatibility lines, the method catalog and the issue history are re-verified.
- `--pipeline-parallel-size` divides the layers into contiguous stages;
`--tensor-parallel-size` still works inside a node, so the canonical two-node
shape is tp = GPUs per node, pp = nodes.
- Two backends: Ray (`--distributed-executor-backend ray`, the worker joining
a `ray start --head`) or multiprocessing (`--distributed-executor-backend mp`
with `--nnodes`, `--node-rank`, `--master-addr`, `--headless` on the
follower). `mp` avoids the Ray dependency; Ray brings the cluster manager.
- What the interconnect must carry: one activation tensor per stage boundary
per forward pass, point to point — not a per-layer reduction. That is the
whole cross-node traffic story; the byte arithmetic and link budgets are
[[Ethernet Bandwidth Math]]. Pin the NIC vLLM talks on
(`NCCL_SOCKET_IFNAME`) rather than letting it guess an interface.
- The PP bubble: with N stages and one sequence in flight, one stage works
while N-1 idle, so pp=2 costs half the machine at batch one. Batching is the
cure — pipeline parallelism "requires many tokens to scale well" — and the
microbatch formula is [[Ethernet Bandwidth Math]] §5.3.
- KV and context: each stage keeps the KV of its own layers, so the pool grows
with the stage count, and `--max-model-len` is not reduced by the split
(recorded 2026-08-14, unverified here).
- What it cannot combine with, in vLLM's words: "Pipeline parallelism is not
composable with speculative decoding as of `vllm<=0.15.0`", and "Speculative
decoding with a draft models is not supported in `vllm<=0.10.0`". Composing
the two has been requested since 2024 and closed stale (#6911, #10615); PP
also carries a Qwen3.5 bug report closed stale without a fix (#36643).
- Method catalog at v0.25.1: EAGLE, MTP, draft model, PARD, MLP, n-gram, suffix
decoding, hidden-state extraction, a custom proposer class, dynamic
speculative decoding; `dflash` is an accepted `method` value.
## Speculative decoding across the split
llama.cpp chains several implementations behind one comma-separated
`--spec-type` list; the split does not change the chain, only where its passes
run. Mechanisms, chaining laws and drafter KV sizing are
[[DFlash DSpark Speculative Decoding]]'s; only the split interaction is here.
| Method | llama.cpp type | Over ggml-rpc | With vLLM PP | Why |
| --- | --- | --- | --- | --- |
| MTP | `draft-mtp` | works; not documented upstream, but run this way on this fleet's split head with acceptance measured ([[Qwen3.8 PP Deployment]]) | no | Context-backed drafter: it needs a draft context and runs in the head process, so RPC devices are simply more devices under it |
| DFlash | `draft-dflash` | same gate, same status as MTP | no | Same draft-context gate (`ctx_dft`); the drafter's KV lands on whichever device the draft context is built on — the one thing to check on a split |
| ngram-mod | `ngram-mod` | safest of the four: no draft model, no draft context, no remote draft pass | no | Draftless — proposals come from cached token n-grams in the head, so the split adds no device work |
| DSpark | `draft-dspark` | **unproven over RPC** — not a missing feature | no | `draft-dspark` exists upstream; nobody has measured it with the target split across two machines |
**Correction (2026-10-19).** A retired stub in this directory asserted "DSpark
is vLLM-only." False. `draft-dspark` is in upstream llama.cpp — parsed in
`common/arg.cpp`, mapped to `COMMON_SPECULATIVE_TYPE_DRAFT_DSPARK` in
`common/speculative.cpp`, where it carries a Markov head and an anchor-first
block layout and aborts with `DSpark draft has no confidence head: please set
--spec-draft-p-min 0` when the drafter has no confidence head. What is unproven
is running it over RPC, and that is the honest limit of the claim.
The vLLM column is uniform because that statement is blanket: it names pipeline
parallelism, not a method, so n-gram and suffix are excluded as written. vLLM
gives no mechanism with it; "the drafter must sit next to the target's layers,
which a stage split separates" is a reading, not upstream's words.
Upstream likewise documents neither for nor against `--rpc` plus
`--spec-type`. What the implementation does say is structural: every
speculative implementation is built in the head process against the head's
device set, and draftless types need no draft context at all. Routing draft
passes to a remote worker, however, has never been tested.
## Mixed-vendor reality
What works today:
- **One vendor per node.** RPC is backend-agnostic by construction: upstream's
own diagram has a host exposing CUDA0+CUDA1, a host exposing Metal, and a
host exposing CUDA0+CPU, all driven by one main host over TCP. Each node
builds its own backend plus `-DGGML_RPC=ON`; nothing speaks CUDA-to-ROCm.
- **Identical build tag on every node**, worker restarted before head. The
verified version-skew gotcha is [[Qwen3.8 PP Deployment]]'s.
- **Speculation on the head** while the split carries the target (table above).
- **RDMA when both peers have it**, auto-negotiated at the handshake; the size
of the gain is third-party and unverified here.
Unproven or broken:
- **One binary holding both CUDA and ROCm for a same-host layer split**:
reported to compile and then list only CUDA devices (issue #11506,
third-party, unverified). Vulkan is the reported single-binary cross-vendor
path, and single-host placement is [[Llama Cpp Multi-Device Splits]]'s, whose
verdict is that per-tensor placement across a vendor mix never works.
- **vLLM across a CUDA node and a ROCm node**: no known experiment; both ends
run the same backend (recorded 2026-08-14). AMD's own two-Halo clustering
guide is tensor parallelism over RCCL, not a mixed-vendor split.
- **DSpark over RPC**: unproven, see the correction above.
- **The exact NVIDIA-desktop + Strix-Halo-NUC pair has no published
benchmark**, and the third-party mixed-vendor figures it is estimated from
(Metal+CUDA over 10 GbE, two Strix Halos over USB4, a three-card CUDA+ROCm set
over 1 GbE, a Vulkan triple in one box, an AMD four-node Strix cluster) live on
[[Expected Performance]] and [[Ethernet Bandwidth Math]], each named with its
model and hardware. None was re-verified in this pass.
## Security
`ggml-rpc-server` has **no authentication and no encryption**. Upstream's own
warning: the backend is "in a proof-of-concept development stage. As such, the
functionality is fragile and insecure. Never run the RPC server on an open
network or in a sensitive environment!" It binds `127.0.0.1:50052` by default —
safe by accident. Bind a routable address and any host that can reach the port
can allocate memory there and supply the tensors it computes on. Trusted LAN
only, firewalled or tunnelled; never a public address.
## Unverified in this pass
Every item was first recorded 2026-08-14, in the research that preceded this
page. Nothing is silently deleted; none is backed by the sources cited below.
- **RPC protocol version and handshake rule** — recorded as protocol `5.0.0`
with "major must match, server minor must be ≥ client". The README's own
banner prints `Starting RPC server v3.0.0`; the conflict is unresolved.
- **RPC internals** — a hard cap of 16 servers, synchronous graph compute, no
tensor copy between two RPC servers, workers loading tensors on first use, a
hash-keyed cache above roughly 10 MB, asserts on quantized tensors whose rows
are not 512-aligned, and `-rpc` on `llama-bench` (the README names only
`llama-cli` and `llama-server`).
- **vLLM flag surface** — `--pipeline-parallel-size`, `--tensor-parallel-size`,
`--distributed-executor-backend {ray,mp}`, `--nnodes`, `--node-rank`,
`--master-addr`, `--headless`, the `ray start --head` join,
`NCCL_SOCKET_IFNAME`.
- **vLLM mechanics** — one activation handoff per stage boundary per forward
pass, a worked ~512 MB handoff for a hidden-5120 model at batch 8 and 2048
positions, `--max-model-len` unchanged by PP, ~50% idle at pp=2 batch one.
- **vLLM throughput verdicts** for a 27B across two nodes over gigabit-class
links; superseded as an estimate by [[Expected Performance]].
- **Mixed-vendor claims** — a CUDA+HIP build listing only CUDA devices
(#11506), Vulkan as the universal cross-vendor backend, AMD's two-Halo
clustering guide being RCCL tensor parallelism, WiFi costing about ten times
wired, the slowest hop setting the decode pace, RDMA worth a few percent.
- **Draft over RPC** — that the draft model stays on the head, and that routing
draft passes to a remote worker has never been confirmed.
## Citations
- [llama.cpp RPC README](https://github.com/ggml-org/llama.cpp/blob/master/tools/rpc/README.md), accessed 2026-10-19 — server usage and device exposure, `-DGGML_RPC=ON` on both sides, the `--rpc host:port` list, proportional weight+KV split and `--tensor-split`, the `-c` local cache, RDMA negotiation and `GGML_RPC_NO_RDMA`, the proof-of-concept and no-authentication warning.
- [llama.cpp multi-GPU guide](https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md), accessed 2026-10-19 — split modes (`-sm layer` = pipeline parallelism, KV with the layer owner), `-ts`/`-ngl`/`-ctk`/`-ctv`/`-fit`, the pipeline-versus-tensor summary, the tensor-mode limits.
- [vLLM speculative decoding docs, v0.25.1](https://docs.vllm.ai/en/v0.25.1/features/speculative_decoding/), accessed 2026-10-19 — the pipeline-parallelism incompatibility statement, the draft-model version floor, the method catalog.
- [llama.cpp `common/speculative.cpp`](https://github.com/ggml-org/llama.cpp/blob/master/common/speculative.cpp), accessed 2026-10-19 — the speculative implementation set including `draft-dspark`, its draft-context gate, its confidence-head error.
- vLLM issue #6911, "[Feature]: Combine pipeline parallelism with speculative decoding", opened 2024-07-29, closed not-planned 2025-03-28, accessed 2026-10-19.
- vLLM issue #10615, "[Usage]: Does speculative decoding support pipeline parallelism?", opened 2024-11-25, closed not-planned 2025-04-18, accessed 2026-10-19.
- vLLM issue #36643, "[Bug]: Qwen3.5 does not work with pipeline parallelism", opened 2026-03-10, closed not-planned 2026-09-24, accessed 2026-10-19.
serverAppConfigs-docs/LLM/Pipeline Parallelism/Sample Script Set.md
+1 -1
View File
@@ -255,7 +255,7 @@ is 30 GiB.
## See Also
- [[Multi-Node Runtimes]]
- [[Qwen3.8 PP Deployment]]
- [[Expected Performance]]
- [[Ethernet Bandwidth Math]]
- [[DFlash DSpark Speculative Decoding]]