`scripts/port-census.py` built its corpus from the new directory, so each page
under test was held by itself: every era page reported 0 orphan lines. With the
corpus excluding the file under test, each candidate holds 14-63 sole-record
lines, so no era page is padding and none was trashed.
Deletions of restatement only:
- Multi-Node Runtimes.md (-257): restated launch material owned by
Qwen3.8 PP Deployment.md; index rows and two Related bullets repointed.
- log.md: 66 headings folded into 35 dated entries (one entry per date, newest
first), titled headings kept as `date — Title` per the spec's wording, and 78
lines of transient deploy chatter dropped where every fact token in the line
is held by an owner page (preserved in .AGENTTRASH/20261019-log-transient/).
- Authentic: the 5-page split reverted (-93); the parent carries
`okf-exempt: length` and the three stub pages stay deleted.
check-okf.py accepts `## YYYY-MM-DD — Title`; port-census.py excludes the file
under test and gained a docstring note about the trap. One-off compaction tool
moved to ~/.AGENTTRASH. Full suite: 451 tests pass, 0 OKF violations,
155 pages, 0 unresolved links.
`Multi-Node Runtimes.md` (+257) restated launch material that
[[Qwen3.8 PP Deployment]] (build target, two-process RPC table, cache-prune
cron), [[Sample Script Set]] (runnable scripts), [[Ethernet Bandwidth Math]]
and [[Expected Performance]] already own, and mixed-vendor numbers owned by
[[x4-pair-experiment]]. Consolidation means the facts end up in ONE place, not
that a new page is written beside the old owners: the four stubs it replaced
were frontmatter-only (9 lines each) and needed no successor. Page trashed,
three inbound links repointed at the real owners.
2026-10-19 09:58:04 -07:00
8 changed files with 53 additions and 383 deletions
description:"How each two-machine runtime is built and launched, what it can combine with, and the mixed-vendor verdicts."
tags:
- pipeline-parallelism
- llamacpp
- vllm
- speculative
timestamp:2026-10-19
---
# Multi-Node Runtimes
Two runtimes can put one model on two machines: llama.cpp over the ggml-rpc
backend, and vLLM with pipeline parallelism. This page owns the cross-runtime
facts — how each is built and launched, what each can and cannot be combined
with, and the mixed-vendor verdicts. Facts owned by a sibling page are linked,
not restated. Bytes-per-token arithmetic and link budgets belong to
[[Ethernet Bandwidth Math]], throughput estimates to [[Expected Performance]],
the real deployment and its verified gotchas to [[Qwen3.8 PP Deployment]], the
educational scripts to [[Sample Script Set]], drafter mechanisms to
[[DFlash DSpark Speculative Decoding]], single-host multi-GPU placement to
[[Llama Cpp Multi-Device Splits]].
## What each runtime is
Both distribute by **layers**, not inside layers.
| | llama.cpp over ggml-rpc | vLLM pipeline parallelism |
| --- | --- | --- |
| unit of distribution | a contiguous slice of layers (`--split-mode layer`, the default) | a contiguous slice of layers (`--pipeline-parallel-size`) |
| process model | one `llama-server`/`llama-cli` on the head; a `ggml-rpc-server` per worker | one `vllm serve` per node, one engine across them |
| what crosses the wire | whatever graph work the head offloads to the remote devices | the hidden state at each stage boundary, point to point |
| KV cache | distributed with the weights; the KV of a layer lives with the owner of that layer | each stage holds the KV of the layers it owns |
| speculation together with the split | yes in llama.cpp (see below) | no, per vLLM's own docs |
Upstream's own summary is why layer-wise is the ethernet choice: pipeline
parallel "minimizes data transfers between GPUs but requires many tokens to
scale well", while tensor parallel "does multiple cross-GPU reductions per
layer … much more bottlenecked by the GPU interconnect speed".
Do not carry the tensor path across machines: `--split-mode tensor` is
experimental, needs flash attention, rejects quantized KV, and is not
implemented for the MoE and hybrid architectures the guide lists — and
per-tensor placement across vendors is not expressible (§1 of [[Llama Cpp Multi-Device Splits]]).
## llama.cpp over ggml-rpc
### Build and launch
Both sides take `-DGGML_RPC=ON` next to that host's own backend:
```bash
cmake .. -DGGML_CUDA=ON -DGGML_RPC=ON # a CUDA worker; -DGGML_HIP=ON for an AMD one
cmake --build . --config Release
```
Worker side — `ggml-rpc-server` exposes ggml devices on the remote host:
```bash
bin/ggml-rpc-server # endpoint 127.0.0.1:50052 by default
bin/ggml-rpc-server --device CUDA0 -c # or CUDA_VISIBLE_DEVICES=0; -c = local cache
```
- It exposes every accelerator it can see; a host with no accelerator exposes a
single `CPU` device. `--device` and `CUDA_VISIBLE_DEVICES` are equivalent
ways to narrow that set.
-`-c` keeps large tensors in `$HOME/.cache/llama.cpp/rpc` (redirect with
`LLAMA_CACHE`) so a big model does not re-stream per load; that cache
filling the worker disk is a recorded fleet incident ([[Qwen3.8 PP Deployment]]).
Head side — build the local backends plus `-DGGML_RPC=ON`, then name the
decoding, hidden-state extraction, a custom proposer class, dynamic
speculative decoding; `dflash` is an accepted `method` value.
## Speculative decoding across the split
llama.cpp chains several implementations behind one comma-separated
`--spec-type` list; the split does not change the chain, only where its passes
run. Mechanisms, chaining laws and drafter KV sizing are
[[DFlash DSpark Speculative Decoding]]'s; only the split interaction is here.
| Method | llama.cpp type | Over ggml-rpc | With vLLM PP | Why |
| --- | --- | --- | --- | --- |
| MTP | `draft-mtp` | works; not documented upstream, but run this way on this fleet's split head with acceptance measured ([[Qwen3.8 PP Deployment]]) | no | Context-backed drafter: it needs a draft context and runs in the head process, so RPC devices are simply more devices under it |
| DFlash | `draft-dflash` | same gate, same status as MTP | no | Same draft-context gate (`ctx_dft`); the drafter's KV lands on whichever device the draft context is built on — the one thing to check on a split |
| ngram-mod | `ngram-mod` | safest of the four: no draft model, no draft context, no remote draft pass | no | Draftless — proposals come from cached token n-grams in the head, so the split adds no device work |
| DSpark | `draft-dspark` | **unproven over RPC** — not a missing feature | no | `draft-dspark` exists upstream; nobody has measured it with the target split across two machines |
**Correction (2026-10-19).** A retired stub in this directory asserted "DSpark
is vLLM-only." False. `draft-dspark` is in upstream llama.cpp — parsed in
`common/arg.cpp`, mapped to `COMMON_SPECULATIVE_TYPE_DRAFT_DSPARK` in
`common/speculative.cpp`, where it carries a Markov head and an anchor-first
block layout and aborts with `DSpark draft has no confidence head: please set
--spec-draft-p-min 0` when the drafter has no confidence head. What is unproven
is running it over RPC, and that is the honest limit of the claim.
The vLLM column is uniform because that statement is blanket: it names pipeline
parallelism, not a method, so n-gram and suffix are excluded as written. vLLM
gives no mechanism with it; "the drafter must sit next to the target's layers,
which a stage split separates" is a reading, not upstream's words.
Upstream likewise documents neither for nor against `--rpc` plus
`--spec-type`. What the implementation does say is structural: every
speculative implementation is built in the head process against the head's
device set, and draftless types need no draft context at all. Routing draft
passes to a remote worker, however, has never been tested.
## Mixed-vendor reality
What works today:
- **One vendor per node.** RPC is backend-agnostic by construction: upstream's
own diagram has a host exposing CUDA0+CUDA1, a host exposing Metal, and a
host exposing CUDA0+CPU, all driven by one main host over TCP. Each node
builds its own backend plus `-DGGML_RPC=ON`; nothing speaks CUDA-to-ROCm.
- **Identical build tag on every node**, worker restarted before head. The
verified version-skew gotcha is [[Qwen3.8 PP Deployment]]'s.
- **Speculation on the head** while the split carries the target (table above).
- **RDMA when both peers have it**, auto-negotiated at the handshake; the size
of the gain is third-party and unverified here.
Unproven or broken:
- **One binary holding both CUDA and ROCm for a same-host layer split**:
reported to compile and then list only CUDA devices (issue #11506,
third-party, unverified). Vulkan is the reported single-binary cross-vendor
path, and single-host placement is [[Llama Cpp Multi-Device Splits]]'s, whose
verdict is that per-tensor placement across a vendor mix never works.
- **vLLM across a CUDA node and a ROCm node**: no known experiment; both ends
run the same backend (recorded 2026-08-14). AMD's own two-Halo clustering
guide is tensor parallelism over RCCL, not a mixed-vendor split.
- **DSpark over RPC**: unproven, see the correction above.
- **The exact NVIDIA-desktop + Strix-Halo-NUC pair has no published
benchmark**, and the third-party mixed-vendor figures it is estimated from
(Metal+CUDA over 10 GbE, two Strix Halos over USB4, a three-card CUDA+ROCm set
over 1 GbE, a Vulkan triple in one box, an AMD four-node Strix cluster) live on
[[Expected Performance]] and [[Ethernet Bandwidth Math]], each named with its
model and hardware. None was re-verified in this pass.
## Security
`ggml-rpc-server` has **no authentication and no encryption**. Upstream's own
warning: the backend is "in a proof-of-concept development stage. As such, the
functionality is fragile and insecure. Never run the RPC server on an open
network or in a sensitive environment!" It binds `127.0.0.1:50052` by default —
safe by accident. Bind a routable address and any host that can reach the port
can allocate memory there and supply the tensors it computes on. Trusted LAN
only, firewalled or tunnelled; never a public address.
## Unverified in this pass
Every item was first recorded 2026-08-14, in the research that preceded this
page. Nothing is silently deleted; none is backed by the sources cited below.
- **RPC protocol version and handshake rule** — recorded as protocol `5.0.0`
with "major must match, server minor must be ≥ client". The README's own
banner prints `Starting RPC server v3.0.0`; the conflict is unresolved.
- **RPC internals** — a hard cap of 16 servers, synchronous graph compute, no
tensor copy between two RPC servers, workers loading tensors on first use, a
`--master-addr`, `--headless`, the `ray start --head` join,
`NCCL_SOCKET_IFNAME`.
- **vLLM mechanics** — one activation handoff per stage boundary per forward
pass, a worked ~512 MB handoff for a hidden-5120 model at batch 8 and 2048
positions, `--max-model-len` unchanged by PP, ~50% idle at pp=2 batch one.
- **vLLM throughput verdicts** for a 27B across two nodes over gigabit-class
links; superseded as an estimate by [[Expected Performance]].
- **Mixed-vendor claims** — a CUDA+HIP build listing only CUDA devices
(#11506), Vulkan as the universal cross-vendor backend, AMD's two-Halo
clustering guide being RCCL tensor parallelism, WiFi costing about ten times
wired, the slowest hop setting the decode pace, RDMA worth a few percent.
- **Draft over RPC** — that the draft model stays on the head, and that routing
draft passes to a remote worker has never been confirmed.
## Citations
- [llama.cpp RPC README](https://github.com/ggml-org/llama.cpp/blob/master/tools/rpc/README.md), accessed 2026-10-19 — server usage and device exposure, `-DGGML_RPC=ON` on both sides, the `--rpc host:port` list, proportional weight+KV split and `--tensor-split`, the `-c` local cache, RDMA negotiation and `GGML_RPC_NO_RDMA`, the proof-of-concept and no-authentication warning.
- [llama.cpp multi-GPU guide](https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md), accessed 2026-10-19 — split modes (`-sm layer` = pipeline parallelism, KV with the layer owner), `-ts`/`-ngl`/`-ctk`/`-ctv`/`-fit`, the pipeline-versus-tensor summary, the tensor-mode limits.
- [vLLM speculative decoding docs, v0.25.1](https://docs.vllm.ai/en/v0.25.1/features/speculative_decoding/), accessed 2026-10-19 — the pipeline-parallelism incompatibility statement, the draft-model version floor, the method catalog.
- [llama.cpp `common/speculative.cpp`](https://github.com/ggml-org/llama.cpp/blob/master/common/speculative.cpp), accessed 2026-10-19 — the speculative implementation set including `draft-dspark`, its draft-context gate, its confidence-head error.
- vLLM issue #6911, "[Feature]: Combine pipeline parallelism with speculative decoding", opened 2024-07-29, closed not-planned 2025-03-28, accessed 2026-10-19.
- vLLM issue #10615, "[Usage]: Does speculative decoding support pipeline parallelism?", opened 2024-11-25, closed not-planned 2025-04-18, accessed 2026-10-19.
- vLLM issue #36643, "[Bug]: Qwen3.5 does not work with pipeline parallelism", opened 2026-03-10, closed not-planned 2026-09-24, accessed 2026-10-19.
`frank-dragoon` at `context_length 262144`, a chat completion carrying
`reasoning_content`, a `tool_calls` response with
`finish_reason: tool_calls`, and SSE streaming.
@@ -2315,9 +2229,6 @@ any repo service.
- **LANDMINE, not changed here:** `/home/dank/models` was cleared by Frank
mid-session to free space. frank-archon is still answering `:7770` from
mmapped-but-unlinked inodes (`/models` is empty inside the running
container) — **the endpoint cannot be restarted until the 175 GiB payload
is re-fetched.** A reboot or a container restart takes frank-archon down.
- Deploy-state reconciliation (Frank's call: no restart) — the autoresearch trio
(`f187165` PLE 1800, `41e9189` prefix caching, `ba2100e` docs) was found already
MERGED to main + pushed AND already live on frank-archon: the running
`frank-archon-r9v` container (booted 12:13 from the `ba2100e` checkout) carries
@@ -2330,9 +2241,7 @@ any repo service.
failure `11-admin` is the FrankBook-Pro Mac mini (`192.168.0.112:8989`) offline
(100 % ICMP loss + TCP timeout), the documented asleep-host condition, not a
regression.
- `scripts/frank-archon.env`, runbook — **prefix caching DEPLOYED on
frank-archon** (Frank's go-ahead; commit `41e9189`): E8-qualified, zero
VRAM cost, 9x/63x reuse TTFT. Verified on :7770 (argv flag, health 200,
completion "PREFIX-CACHE-LIVE"). Gotcha re-confirmed: the container is
in dank's ROOTLESS podman store (`sudo podman` inspects an empty store
and reports nothing); probe via the shim's `docker` or the unit MainPID.
@@ -2464,7 +2373,6 @@ any repo service.
## 2026-08-26
-`/dankclaw/Caddyfile` — Both Plex vhosts proxy to PMS over a TLS upstream (`insecure_skip_verify`); PMS drops plain-HTTP connections from non-loopback sources (empty-body 502s).
-`/dankclaw/gitea/quadlet/gitea-caddy.container` — Standalone Gitea Caddy quadlet retired; web proxy ownership moved to canonical Caddy; deploy-plan/deploy.sh/deploy-remote.sh + tests updated for the migration.
- Incident — Post-reboot: Caddy (a bare compose container — `restart: unless-stopped` does not survive shutdown's explicit stop) and Pi-hole (quadlet crash-loop, restart counter 1645) were both down. Root cause: `net.ipv4.ip_unprivileged_port_start` reset to 80 by a later-sorted sysctl drop-in AND a stale line in `/etc/sysctl.conf`, which systemd-sysctl applies AFTER all of `sysctl.d/`.
-`/AGENTS.md` — New MANDATORY "Boot Persistence & Self-Healing" section.
-`/dankclaw/caddy/quadlet/caddy.container`, `/dankclaw/caddy/env.example` — Caddy is now a boot-persistent quadlet (joins `dankclaw_default`, `Pull=never`, env from `/home/dank/Containers/caddy/env`); removed from compose.yaml.
-`/dankclaw/pihole/sysctl/99-dnsmasq.conf` — Repo source of truth for `ip_unprivileged_port_start=53`; `deploy_pihole` stages conflicting drop-ins, comments out the stale sysctl.conf line, and applies `sysctl --system` (the boot code path).
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.