docs(vault): one fact, one owner — repair the doubled pages, single-source the facts

Structural repair:
- 13 pages were the same document twice (each exactly 2x its HEAD size),
  a second frontmatter block whose closing fence had been glued onto a body
  line by a frontmatter rewrite that re-read its own output. Un-doubled by
  cutting at the second frontmatter and keeping the copy whose body is a
  superset; the orphan census reports 0 lost tokens against HEAD.
- log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first —
  66 dated headings merged into 34, titles demoted to bold leads, every
  content line verified present. The file had been descending at the head
  and ascending at the tail.

Ownership moves:
- New `LLM/frank-medium.md` is the locator for the retired Dankserver
  endpoint: era windows quoted from each era page's own banner, no numbers
  restated, and the two unresolved date conflicts recorded instead of
  silently resolved. `frank-archon.md` links to it instead of restating the
  pre-rename window.
- The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone;
  `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor
  verdicts and cross-split spec decode, and the four links that pointed at
  the stubs were repointed at it or at the drafter page.
- Authentik's three stub pages fold into `Authentik.md`, with the process
  detail and the outpost reference split out so the parent stays under the
  length hint.

Verified corrections (host census + live probe, 2026-10-19):
- The MXFP4 checkpoint two pages called the "on-disk rollback" is gone:
  /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on
  the era page that owns that payload; the swap page now links to it.
- frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2
  arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run
  census is kept once (155 requests) instead of twice (~120 and 155).
- Fleet shapes re-confirmed against the live endpoints before touching any
  page: 3x200000 fp8 KV, 2x200704, 2x262144.

Index rows are regenerated, not hand-carved. Net -1,849 vault lines.

serverAppConfigs-docs/LLM/Pipeline Parallelism/Ethernet Bandwidth Math.md

@@ -1,11 +1,11 @@
 ---
 type: reference
 title: Ethernet Bandwidth Math
-description: Bytes-per-token math for pipeline vs tensor parallelism, feasibility tables for 1/2.5/10 GbE, and real multi-node benchmark numbers.
+description: "Bytes-per-token math for pipeline versus tensor parallelism, and the link budgets that follow from it."
 tags:
   - pipeline-parallelism
   - research
-timestamp: 2026-08-17
+timestamp: 2026-10-19
 ---
 
 # Bandwidth Analysis: Multi-Node LLM Inference over Ethernet
@@ -147,19 +147,11 @@ bytes_per_boundary = batch_size × hidden_size × 2   [fp16 = 2 bytes]
 - Per-token, per-boundary: `6144 × 2 = 12,288 bytes = 12 KB`
 - With Nₚₚ stage boundaries: total = `Nₚₚ × 12 KB/tok`
 
-**Full example — 27B model, 2 PP stages (1 boundary), batch=1:**
+**Worked examples (1 boundary = 2 PP stages):**
 ```
-1 boundary × 4096 × 2 bytes = 8,192 bytes/tok
-```
-
-**Example — same model, batch=64:**
-```
-1 boundary × 4096 × 64 × 2 bytes = 524,288 bytes = 512 KB/tok
-```
-
-**Example — 36B model, 2 PP stages, batch=1:**
-```
-1 boundary × 6144 × 2 bytes = 12,288 bytes = 12 KB/tok
+27B, batch=1:  1 boundary × 4096 × 2      bytes = 8,192 bytes/tok  = 8 KB
+27B, batch=64: 1 boundary × 4096 × 64 × 2 bytes = 524,288          = 512 KB/tok
+36B, batch=1:  1 boundary × 6144 × 2      bytes = 12,288 bytes/tok = 12 KB
 ```
 
 ### 1.2 Tensor Parallelism (TP) — for comparison
@@ -244,57 +236,13 @@ total_bytes_per_tok = L × batch_size × hidden_size × 2   [L all-reduces total
 | 50 | 10.24 Mbit/s | 163.84 Mbit/s | 655.36 Mbit/s |
 | 100 | 20.48 Mbit/s | 327.68 Mbit/s | 1,310.72 Mbit/s |
 
-### 2.3 Feasibility matrix
+### 2.3 Feasibility
 
-Green = feasible (< 50% of link capacity). Yellow = marginal (50–90%). Red = saturated (> 90%).
-
-**27B model — PP (1 boundary), batch=1:**
-
-| | 1 GbE (118 MB/s) | 2.5 GbE (294 MB/s) | 10 GbE (1175 MB/s) |
-|---|---|---|---|
-| 10 tok/s | ✅ 0.13 Mbit/s | ✅ 0.13 Mbit/s | ✅ 0.13 Mbit/s |
-| 50 tok/s | ✅ 0.65 Mbit/s | ✅ 0.65 Mbit/s | ✅ 0.65 Mbit/s |
-| 100 tok/s | ✅ 1.31 Mbit/s | ✅ 1.31 Mbit/s | ✅ 1.31 Mbit/s |
-
-**27B model — PP (1 boundary), batch=16:**
-
-| | 1 GbE | 2.5 GbE | 10 GbE |
-|---|---|---|---|
-| 10 tok/s | ✅ 2.05 Mbit/s | ✅ 2.05 Mbit/s | ✅ 2.05 Mbit/s |
-| 50 tok/s | ✅ 10.24 Mbit/s | ✅ 10.24 Mbit/s | ✅ 10.24 Mbit/s |
-| 100 tok/s | ✅ 20.48 Mbit/s | ✅ 20.48 Mbit/s | ✅ 20.48 Mbit/s |
-
-**27B model — PP (1 boundary), batch=64:**
-
-| | 1 GbE | 2.5 GbE | 10 GbE |
-|---|---|---|---|
-| 10 tok/s | ✅ 8.19 Mbit/s | ✅ 8.19 Mbit/s | ✅ 8.19 Mbit/s |
-| 50 tok/s | ✅ 40.96 Mbit/s | ✅ 40.96 Mbit/s | ✅ 40.96 Mbit/s |
-| 100 tok/s | ✅ 81.92 Mbit/s | ✅ 81.92 Mbit/s | ✅ 81.92 Mbit/s |
-
-**27B model — TP (32 layers), batch=1:**
-
-| | 1 GbE | 2.5 GbE | 10 GbE |
-|---|---|---|---|
-| 10 tok/s | ✅ 2.05 Mbit/s | ✅ 2.05 Mbit/s | ✅ 2.05 Mbit/s |
-| 50 tok/s | ✅ 10.24 Mbit/s | ✅ 10.24 Mbit/s | ✅ 10.24 Mbit/s |
-| 100 tok/s | ✅ 20.48 Mbit/s | ✅ 20.48 Mbit/s | ✅ 20.48 Mbit/s |
-
-**27B model — TP (32 layers), batch=16:**
-
-| | 1 GbE | 2.5 GbE | 10 GbE |
-|---|---|---|---|
-| 10 tok/s | ✅ 32.77 Mbit/s | ✅ 32.77 Mbit/s | ✅ 32.77 Mbit/s |
-| 50 tok/s | ✅ 163.84 Mbit/s | ✅ 163.84 Mbit/s | ✅ 163.84 Mbit/s |
-| 100 tok/s | ✅ 327.68 Mbit/s | ✅ 327.68 Mbit/s | ✅ 327.68 Mbit/s |
-
-**27B model — TP (32 layers), batch=64:**
-
-| | 1 GbE | 2.5 GbE | 10 GbE |
-|---|---|---|---|
-| 10 tok/s | ✅ 131.07 Mbit/s | ✅ 131.07 Mbit/s | ✅ 131.07 Mbit/s |
-| 50 tok/s | ✅ 655.36 Mbit/s | ✅ 655.36 Mbit/s | ✅ 655.36 Mbit/s |
-| 100 tok/s | ✅ 1310.72 Mbit/s | ✅ 1310.72 Mbit/s | ✅ 1310.72 Mbit/s |
+Thresholds: **feasible** = under 50% of link capacity, **marginal** = 50–90%,
+**saturated** = over 90%. Cross-checking §2.2 against §2.1: every PP case is
+feasible on every link class — even the heaviest, 27B at batch=64 and 100
+tok/s (81.92 Mbit/s), is feasible on 1 GbE — and every TP case is feasible
+except batch=64:
 
 **27B TP batch=64 at 100 tok/s: 1310.72 Mbit/s > 1 GbE usable (1000 Mbit/s) — link-saturated on 1 GbE**
 
@@ -368,10 +316,7 @@ With TP, each layer transfers `H × 2` bytes but this happens L times. For 256 K
 **Petals [1]:**
 - Uses pipeline parallelism with dynamic 8-bit quantization (halves bandwidth)
 - For BLOOM-176B (H=14336), per-boundary communication is ~14 KB/tok with quantization
-- Recommended bandwidth: 25 Mbit/s bidirectional for interactive inference [1]
 - Key finding: "For inference, performance does not depend much on bandwidth or sequence length but degrades with higher latency" — RTT matters more than throughput
-- At 1 Gbit/s with <5 ms RTT: 1.71 steps/s; at 100 Mbit/s with <5 ms: 1.66 steps/s (only 3% difference)
-- At 100 Mbit/s with 100 ms RTT: 1.23 steps/s (28% degradation)
 
 **llama.cpp RPC [2]:**
 - Model weights and KV cache distributed across nodes proportionally to memory
@@ -493,38 +438,17 @@ With Nₚₚ = 4 and 30 µs per-hop RTT: ~120 µs network, still <6% of compute
 
 ## Appendix: Full bandwidth formulae
 
-### Pipeline Parallelism (generation, batch=1)
+### Pipeline and Tensor Parallelism (generation)
+
+`N` is the batch size; `N = 1` is the batch=1 form.
 
 ```
-bytes_per_boundary_per_tok = hidden_size × 2   [fp16]
-total_bytes_per_tok = num_boundaries × hidden_size × 2
+bytes_per_boundary_per_tok = hidden_size × N × 2              [fp16, PP boundary]
+bytes_per_layer_allreduce  = hidden_size × N × 2              [TP, one all-reduce per layer]
+total_bytes_per_tok = num_boundaries × hidden_size × N × 2    [PP]
+total_bytes_per_tok = num_layers      × hidden_size × N × 2   [TP]
 bandwidth_mbps = total_bytes_per_tok × tokens_per_second × 8 / 1,000,000
-```
-
-### Pipeline Parallelism (generation, batch=N)
 
-```
-bytes_per_boundary_per_tok = hidden_size × N × 2
-total_bytes_per_tok = num_boundaries × hidden_size × N × 2
-```
-
-### Tensor Parallelism (generation, batch=1)
-
-```
-bytes_per_layer_allreduce = hidden_size × 2
-total_bytes_per_tok = num_layers × hidden_size × 2
-bandwidth_mbps = total_bytes_per_tok × tokens_per_second × 8 / 1,000,000
-```
-
-### Tensor Parallelism (generation, batch=N)
-
-```
-total_bytes_per_tok = num_layers × hidden_size × N × 2
-```
-
-### With 8-bit quantization (like Petals)
-
-```
-# Simply divide all fp16 byte counts by 2
-bytes_per_boundary_per_tok = hidden_size × 1   [int8]
+# With 8-bit quantization (like Petals): simply divide all fp16 byte counts by 2
+bytes_per_boundary_per_tok = hidden_size × 1                  [int8]
 ```

serverAppConfigs-docs/LLM/Pipeline Parallelism/Expected Performance.md

@@ -1,7 +1,7 @@
 ---
 type: reference
 title: Expected Performance
-description: Estimated tokens per second for a 25-40B model split across two nodes over 1/2.5/10 GbE, with real benchmark anchors.
+description: "Estimated split-model throughput over ethernet, with the benchmark anchors the estimates rest on."
 tags:
   - pipeline-parallelism
   - performance
@@ -13,7 +13,7 @@ timestamp: 2026-08-14
 
 This page estimates the speed of a 25-40B model split across two machines
 over ethernet. It uses the bandwidth math in [[Ethernet Bandwidth Math]]
-and the measured numbers in [[Mixed-Vendor Experiments]].
+and the mixed-vendor measurements in [[Multi-Node Runtimes]].
 
 ## The Bandwidth Ceiling Is Not the Limit
 
@@ -93,5 +93,5 @@ roughly 200k-260k tokens across both nodes.
 ## See Also
 
 - [[Ethernet Bandwidth Math]]
-- [[Mixed-Vendor Experiments]]
+- [[Multi-Node Runtimes]]
 - [[Sample Script Set]]

serverAppConfigs-docs/LLM/Pipeline Parallelism/Mixed-Vendor Experiments.md

@@ -1,9 +0,0 @@
----
-type: reference
-title: "Mixed-Vendor Experiments"
-description: "Real-world NVIDIA + AMD Strix Halo multi-node experiments: what works (llama.cpp RPC, Vulkan split), what breaks (mixed CUDA+ROCm vLLM), measured tok/s."
-tags:
-  - pipeline-parallelism
-  - research
-timestamp: 2026-08-14
----

serverAppConfigs-docs/LLM/Pipeline Parallelism/Multi-Node Runtimes.md

@@ -0,0 +1,257 @@
+---
+type: reference
+title: Multi-Node Runtimes
+description: "How each two-machine runtime is built and launched, what it can combine with, and the mixed-vendor verdicts."
+tags:
+  - pipeline-parallelism
+  - llamacpp
+  - vllm
+  - speculative
+timestamp: 2026-10-19
+---
+
+# Multi-Node Runtimes
+
+Two runtimes can put one model on two machines: llama.cpp over the ggml-rpc
+backend, and vLLM with pipeline parallelism. This page owns the cross-runtime
+facts — how each is built and launched, what each can and cannot be combined
+with, and the mixed-vendor verdicts. Facts owned by a sibling page are linked,
+not restated. Bytes-per-token arithmetic and link budgets belong to
+[[Ethernet Bandwidth Math]], throughput estimates to [[Expected Performance]],
+the real deployment and its verified gotchas to [[Qwen3.8 PP Deployment]], the
+educational scripts to [[Sample Script Set]], drafter mechanisms to
+[[DFlash DSpark Speculative Decoding]], single-host multi-GPU placement to
+[[Llama Cpp Multi-Device Splits]].
+
+## What each runtime is
+
+Both distribute by **layers**, not inside layers.
+
+| | llama.cpp over ggml-rpc | vLLM pipeline parallelism |
+| --- | --- | --- |
+| unit of distribution | a contiguous slice of layers (`--split-mode layer`, the default) | a contiguous slice of layers (`--pipeline-parallel-size`) |
+| process model | one `llama-server`/`llama-cli` on the head; a `ggml-rpc-server` per worker | one `vllm serve` per node, one engine across them |
+| what crosses the wire | whatever graph work the head offloads to the remote devices | the hidden state at each stage boundary, point to point |
+| KV cache | distributed with the weights; the KV of a layer lives with the owner of that layer | each stage holds the KV of the layers it owns |
+| speculation together with the split | yes in llama.cpp (see below) | no, per vLLM's own docs |
+
+Upstream's own summary is why layer-wise is the ethernet choice: pipeline
+parallel "minimizes data transfers between GPUs but requires many tokens to
+scale well", while tensor parallel "does multiple cross-GPU reductions per
+layer … much more bottlenecked by the GPU interconnect speed".
+
+Do not carry the tensor path across machines: `--split-mode tensor` is
+experimental, needs flash attention, rejects quantized KV, and is not
+implemented for the MoE and hybrid architectures the guide lists — and
+per-tensor placement across vendors is not expressible (§1 of [[Llama Cpp Multi-Device Splits]]).
+
+## llama.cpp over ggml-rpc
+
+### Build and launch
+
+Both sides take `-DGGML_RPC=ON` next to that host's own backend:
+
+```bash
+cmake .. -DGGML_CUDA=ON -DGGML_RPC=ON    # a CUDA worker; -DGGML_HIP=ON for an AMD one
+cmake --build . --config Release
+```
+
+Worker side — `ggml-rpc-server` exposes ggml devices on the remote host:
+
+```bash
+bin/ggml-rpc-server                      # endpoint 127.0.0.1:50052 by default
+bin/ggml-rpc-server --device CUDA0 -c    # or CUDA_VISIBLE_DEVICES=0; -c = local cache
+```
+
+- It exposes every accelerator it can see; a host with no accelerator exposes a
+  single `CPU` device. `--device` and `CUDA_VISIBLE_DEVICES` are equivalent
+  ways to narrow that set.
+- `-c` keeps large tensors in `$HOME/.cache/llama.cpp/rpc` (redirect with
+  `LLAMA_CACHE`) so a big model does not re-stream per load; that cache
+  filling the worker disk is a recorded fleet incident ([[Qwen3.8 PP Deployment]]).
+
+Head side — build the local backends plus `-DGGML_RPC=ON`, then name the
+workers:
+
+```bash
+llama-server -m model.gguf --rpc 192.168.88.10:50052,192.168.88.11:50052 -ngl 99
+```
+
+`--rpc` is a comma-separated list of `host:port`, one entry per
+`ggml-rpc-server`; the head's own devices stay in the pool next to the remote
+ones. Launch scripts for this shape: [[Sample Script Set]].
+
+### Split, context, KV
+
+- Default placement: weights **and** KV across all devices, local and remote,
+  "in proportion to each device's available memory". Override with
+  `--tensor-split`, whose proportions follow the order in `--device`. Why the
+  fleet always pins it (a Strix reports a huge GTT window, so the automatic
+  split lands on the slow device) is [[Qwen3.8 PP Deployment]]'s.
+- `-sm layer` (the default) is pipeline parallelism: each device holds a
+  contiguous slice of layers and the KV cache of a layer lives on the device
+  that owns that layer.
+- KV types are the usual `-ctk` / `-ctv`, `f16` by default. `--fit` is on by
+  default and sizes unset arguments to device memory; turn it off to hold a
+  context you set on purpose. Context is one global `--ctx-size` — the RPC
+  README documents no per-node context or per-node KV knob.
+- Upstream's out-of-memory ladder (stated for tensor mode, useful order
+  anyway): lower `--ctx-size`, then `--parallel`, then `--n-gpu-layers`.
+- Transport is TCP; RDMA is negotiated at the handshake with no flag changes
+  and falls back to TCP unless both peers can use it (Linux RoCEv2 via
+  `libibverbs`, macOS RDMA over Thunderbolt 5 via `librdma`). It is point to
+  point — name the peer's RDMA-capable address in `--rpc` or the session stays
+  on TCP. `GGML_RPC_NO_RDMA=1` forces TCP; `GGML_RPC_DEBUG` traces the backend.
+
+Upstream calls the backend a proof of concept, "fragile and insecure" (see the
+Security section). Two limits shape the design: decode is bound by the per-step
+round trip, not by the link's bytes ([[Ethernet Bandwidth Math]]), and every
+node must run the identical build or the handshake hangs ([[Qwen3.8 PP Deployment]]).
+
+## vLLM pipeline parallelism
+
+> **Provenance.** Flags and mechanics below were recorded 2026-08-14 and are
+> not re-checked upstream in this pass (see "Unverified in this pass"); the
+> incompatibility lines, the method catalog and the issue history are re-verified.
+
+- `--pipeline-parallel-size` divides the layers into contiguous stages;
+  `--tensor-parallel-size` still works inside a node, so the canonical two-node
+  shape is tp = GPUs per node, pp = nodes.
+- Two backends: Ray (`--distributed-executor-backend ray`, the worker joining
+  a `ray start --head`) or multiprocessing (`--distributed-executor-backend mp`
+  with `--nnodes`, `--node-rank`, `--master-addr`, `--headless` on the
+  follower). `mp` avoids the Ray dependency; Ray brings the cluster manager.
+- What the interconnect must carry: one activation tensor per stage boundary
+  per forward pass, point to point — not a per-layer reduction. That is the
+  whole cross-node traffic story; the byte arithmetic and link budgets are
+  [[Ethernet Bandwidth Math]]. Pin the NIC vLLM talks on
+  (`NCCL_SOCKET_IFNAME`) rather than letting it guess an interface.
+- The PP bubble: with N stages and one sequence in flight, one stage works
+  while N-1 idle, so pp=2 costs half the machine at batch one. Batching is the
+  cure — pipeline parallelism "requires many tokens to scale well" — and the
+  microbatch formula is [[Ethernet Bandwidth Math]] §5.3.
+- KV and context: each stage keeps the KV of its own layers, so the pool grows
+  with the stage count, and `--max-model-len` is not reduced by the split
+  (recorded 2026-08-14, unverified here).
+- What it cannot combine with, in vLLM's words: "Pipeline parallelism is not
+  composable with speculative decoding as of `vllm<=0.15.0`", and "Speculative
+  decoding with a draft models is not supported in `vllm<=0.10.0`". Composing
+  the two has been requested since 2024 and closed stale (#6911, #10615); PP
+  also carries a Qwen3.5 bug report closed stale without a fix (#36643).
+- Method catalog at v0.25.1: EAGLE, MTP, draft model, PARD, MLP, n-gram, suffix
+  decoding, hidden-state extraction, a custom proposer class, dynamic
+  speculative decoding; `dflash` is an accepted `method` value.
+
+## Speculative decoding across the split
+
+llama.cpp chains several implementations behind one comma-separated
+`--spec-type` list; the split does not change the chain, only where its passes
+run. Mechanisms, chaining laws and drafter KV sizing are
+[[DFlash DSpark Speculative Decoding]]'s; only the split interaction is here.
+
+| Method | llama.cpp type | Over ggml-rpc | With vLLM PP | Why |
+| --- | --- | --- | --- | --- |
+| MTP | `draft-mtp` | works; not documented upstream, but run this way on this fleet's split head with acceptance measured ([[Qwen3.8 PP Deployment]]) | no | Context-backed drafter: it needs a draft context and runs in the head process, so RPC devices are simply more devices under it |
+| DFlash | `draft-dflash` | same gate, same status as MTP | no | Same draft-context gate (`ctx_dft`); the drafter's KV lands on whichever device the draft context is built on — the one thing to check on a split |
+| ngram-mod | `ngram-mod` | safest of the four: no draft model, no draft context, no remote draft pass | no | Draftless — proposals come from cached token n-grams in the head, so the split adds no device work |
+| DSpark | `draft-dspark` | **unproven over RPC** — not a missing feature | no | `draft-dspark` exists upstream; nobody has measured it with the target split across two machines |
+
+**Correction (2026-10-19).** A retired stub in this directory asserted "DSpark
+is vLLM-only." False. `draft-dspark` is in upstream llama.cpp — parsed in
+`common/arg.cpp`, mapped to `COMMON_SPECULATIVE_TYPE_DRAFT_DSPARK` in
+`common/speculative.cpp`, where it carries a Markov head and an anchor-first
+block layout and aborts with `DSpark draft has no confidence head: please set
+--spec-draft-p-min 0` when the drafter has no confidence head. What is unproven
+is running it over RPC, and that is the honest limit of the claim.
+
+The vLLM column is uniform because that statement is blanket: it names pipeline
+parallelism, not a method, so n-gram and suffix are excluded as written. vLLM
+gives no mechanism with it; "the drafter must sit next to the target's layers,
+which a stage split separates" is a reading, not upstream's words.
+
+Upstream likewise documents neither for nor against `--rpc` plus
+`--spec-type`. What the implementation does say is structural: every
+speculative implementation is built in the head process against the head's
+device set, and draftless types need no draft context at all. Routing draft
+passes to a remote worker, however, has never been tested.
+
+## Mixed-vendor reality
+
+What works today:
+
+- **One vendor per node.** RPC is backend-agnostic by construction: upstream's
+  own diagram has a host exposing CUDA0+CUDA1, a host exposing Metal, and a
+  host exposing CUDA0+CPU, all driven by one main host over TCP. Each node
+  builds its own backend plus `-DGGML_RPC=ON`; nothing speaks CUDA-to-ROCm.
+- **Identical build tag on every node**, worker restarted before head. The
+  verified version-skew gotcha is [[Qwen3.8 PP Deployment]]'s.
+- **Speculation on the head** while the split carries the target (table above).
+- **RDMA when both peers have it**, auto-negotiated at the handshake; the size
+  of the gain is third-party and unverified here.
+
+Unproven or broken:
+
+- **One binary holding both CUDA and ROCm for a same-host layer split**:
+  reported to compile and then list only CUDA devices (issue #11506,
+  third-party, unverified). Vulkan is the reported single-binary cross-vendor
+  path, and single-host placement is [[Llama Cpp Multi-Device Splits]]'s, whose
+  verdict is that per-tensor placement across a vendor mix never works.
+- **vLLM across a CUDA node and a ROCm node**: no known experiment; both ends
+  run the same backend (recorded 2026-08-14). AMD's own two-Halo clustering
+  guide is tensor parallelism over RCCL, not a mixed-vendor split.
+- **DSpark over RPC**: unproven, see the correction above.
+- **The exact NVIDIA-desktop + Strix-Halo-NUC pair has no published
+  benchmark**, and the third-party mixed-vendor figures it is estimated from
+  (Metal+CUDA over 10 GbE, two Strix Halos over USB4, a three-card CUDA+ROCm set
+  over 1 GbE, a Vulkan triple in one box, an AMD four-node Strix cluster) live on
+  [[Expected Performance]] and [[Ethernet Bandwidth Math]], each named with its
+  model and hardware. None was re-verified in this pass.
+
+## Security
+
+`ggml-rpc-server` has **no authentication and no encryption**. Upstream's own
+warning: the backend is "in a proof-of-concept development stage. As such, the
+functionality is fragile and insecure. Never run the RPC server on an open
+network or in a sensitive environment!" It binds `127.0.0.1:50052` by default —
+safe by accident. Bind a routable address and any host that can reach the port
+can allocate memory there and supply the tensors it computes on. Trusted LAN
+only, firewalled or tunnelled; never a public address.
+
+## Unverified in this pass
+
+Every item was first recorded 2026-08-14, in the research that preceded this
+page. Nothing is silently deleted; none is backed by the sources cited below.
+
+- **RPC protocol version and handshake rule** — recorded as protocol `5.0.0`
+  with "major must match, server minor must be ≥ client". The README's own
+  banner prints `Starting RPC server v3.0.0`; the conflict is unresolved.
+- **RPC internals** — a hard cap of 16 servers, synchronous graph compute, no
+  tensor copy between two RPC servers, workers loading tensors on first use, a
+  hash-keyed cache above roughly 10 MB, asserts on quantized tensors whose rows
+  are not 512-aligned, and `-rpc` on `llama-bench` (the README names only
+  `llama-cli` and `llama-server`).
+- **vLLM flag surface** — `--pipeline-parallel-size`, `--tensor-parallel-size`,
+  `--distributed-executor-backend {ray,mp}`, `--nnodes`, `--node-rank`,
+  `--master-addr`, `--headless`, the `ray start --head` join,
+  `NCCL_SOCKET_IFNAME`.
+- **vLLM mechanics** — one activation handoff per stage boundary per forward
+  pass, a worked ~512 MB handoff for a hidden-5120 model at batch 8 and 2048
+  positions, `--max-model-len` unchanged by PP, ~50% idle at pp=2 batch one.
+- **vLLM throughput verdicts** for a 27B across two nodes over gigabit-class
+  links; superseded as an estimate by [[Expected Performance]].
+- **Mixed-vendor claims** — a CUDA+HIP build listing only CUDA devices
+  (#11506), Vulkan as the universal cross-vendor backend, AMD's two-Halo
+  clustering guide being RCCL tensor parallelism, WiFi costing about ten times
+  wired, the slowest hop setting the decode pace, RDMA worth a few percent.
+- **Draft over RPC** — that the draft model stays on the head, and that routing
+  draft passes to a remote worker has never been confirmed.
+
+## Citations
+
+- [llama.cpp RPC README](https://github.com/ggml-org/llama.cpp/blob/master/tools/rpc/README.md), accessed 2026-10-19 — server usage and device exposure, `-DGGML_RPC=ON` on both sides, the `--rpc host:port` list, proportional weight+KV split and `--tensor-split`, the `-c` local cache, RDMA negotiation and `GGML_RPC_NO_RDMA`, the proof-of-concept and no-authentication warning.
+- [llama.cpp multi-GPU guide](https://github.com/ggml-org/llama.cpp/blob/master/docs/multi-gpu.md), accessed 2026-10-19 — split modes (`-sm layer` = pipeline parallelism, KV with the layer owner), `-ts`/`-ngl`/`-ctk`/`-ctv`/`-fit`, the pipeline-versus-tensor summary, the tensor-mode limits.
+- [vLLM speculative decoding docs, v0.25.1](https://docs.vllm.ai/en/v0.25.1/features/speculative_decoding/), accessed 2026-10-19 — the pipeline-parallelism incompatibility statement, the draft-model version floor, the method catalog.
+- [llama.cpp `common/speculative.cpp`](https://github.com/ggml-org/llama.cpp/blob/master/common/speculative.cpp), accessed 2026-10-19 — the speculative implementation set including `draft-dspark`, its draft-context gate, its confidence-head error.
+- vLLM issue #6911, "[Feature]: Combine pipeline parallelism with speculative decoding", opened 2024-07-29, closed not-planned 2025-03-28, accessed 2026-10-19.
+- vLLM issue #10615, "[Usage]: Does speculative decoding support pipeline parallelism?", opened 2024-11-25, closed not-planned 2025-04-18, accessed 2026-10-19.
+- vLLM issue #36643, "[Bug]: Qwen3.5 does not work with pipeline parallelism", opened 2026-03-10, closed not-planned 2026-09-24, accessed 2026-10-19.

serverAppConfigs-docs/LLM/Pipeline Parallelism/Qwen3.6 27B Spec Methods.md

@@ -1,9 +0,0 @@
----
-type: reference
-title: "Qwen3.6 27B Spec Methods"
-description: "Qwen3.6-27B Q8 GGUF variants and llama.cpp speculative decoding (draft-mtp, draft-dflash, ngram-mod): flags, RPC compatibility, memory/context math."
-tags:
-  - pipeline-parallelism
-  - research
-timestamp: 2026-08-14
----

serverAppConfigs-docs/LLM/Pipeline Parallelism/Qwen3.8 PP Deployment.md

@@ -1,6 +1,6 @@
 ---
 type: reference
-description: "Split Qwen3.8-27B across the Strix Halo iGPU (head) and the RTX"
+description: "The real split deployment: which node is head and which is worker, the flags, and the verified gotchas."
 tags:
   - llm
   - pipeline

serverAppConfigs-docs/LLM/Pipeline Parallelism/Sample Script Set.md

@@ -1,7 +1,7 @@
 ---
 type: reference
 title: Sample Script Set
-description: An educational two-node llama.cpp RPC split of Qwen3.6-27B-Q8_K_P over ethernet. For reading only. Do not run or deploy.
+description: "An educational two-node RPC split script set, for reading only — never run it or deploy it."
 tags:
   - pipeline-parallelism
   - sample
@@ -255,7 +255,7 @@ is 30 GiB.
 
 ## See Also
 
-- [[llama.cpp RPC]]
+- [[Multi-Node Runtimes]]
 - [[Expected Performance]]
 - [[Ethernet Bandwidth Math]]
-- [[Qwen3.6 27B Spec Methods]]
+- [[DFlash DSpark Speculative Decoding]]

serverAppConfigs-docs/LLM/Pipeline Parallelism/index.md

@@ -28,28 +28,24 @@ graph LR
     end
 ```
 
-- [[llama.cpp RPC]] — the working, production-capable path for two
   heterogeneous nodes over ethernet.
-- [[vLLM Pipeline Parallelism]] — supported but heavier; needs Ray or
   multiprocessing and does not compose with speculative decoding yet.
 
 ## What the Research Found
 
-- [[Ethernet Bandwidth Math]] — a PP boundary moves only the hidden state
+- [[Ethernet Bandwidth Math]] — Bytes-per-token math for pipeline versus tensor parallelism, and the link budgets that follow from it.
   (about 10 KiB per token for a 27B model). 1 GbE supports hundreds of
   tokens per second on paper; the real limit is latency and RPC overhead.
-- [[Mixed-Vendor Experiments]] — NVIDIA + Strix Halo multi-node inference
   works today via llama.cpp RPC with separate CUDA and ROCm builds. The
   exact NVIDIA-desktop + Strix-Halo-NUC pair has not been benchmarked with
   published numbers yet.
-- [[Qwen3.6 27B Spec Methods]] — Q8 GGUF variants, and which speculative
   methods (MTP, DFlash, ngram-mod) work with RPC. DSpark is vLLM-only.
-- [[Expected Performance]] — estimated tokens per second for a 27B Q8 split
+- [[Expected Performance]] — Estimated split-model throughput over ethernet, with the benchmark anchors the estimates rest on.
   across an RTX 3090 and a Strix Halo NUC over 1/2.5/10 GbE.
 
 ## Deployment
 
-- [[Qwen3.8 PP Deployment]] — the real Qwen3.8-27B split: head on Dankclaw
+- [[Qwen3.8 PP Deployment]] — The real split deployment: which node is head and which is worker, the flags, and the verified gotchas.
   (Strix), worker on Dankserver (RTX 3090), b10433 on both sides, scripts in
   `scripts/`, morning-of test runbook. Verified gotchas: no `--device` with
   `--rpc`, default split is proportional to reported free memory (pin 1,1),
@@ -57,7 +53,7 @@ graph LR
 
 ## Sample Scripts
 
-- [[Sample Script Set]] — a two-node llama.cpp RPC split of
+- [[Sample Script Set]] — An educational two-node RPC split script set, for reading only — never run it or deploy it.
   Qwen3.6-27B-Q8_K_P with MTP, DFlash, and ngram-mod variants, sized for
   the largest context that fits. For educational reading only. Do not run
   it in production.
@@ -68,3 +64,9 @@ graph LR
 - [[frank-zealot]]
 - [[Dankserver System Specs]]
 - [[Dankclaw System Specs]]
+
+## Pages
+
+| Page | Owns |
+|---|---|
+| [[Multi-Node Runtimes]] | How each two-machine runtime is built and launched, what it can combine with, and the mixed-vendor verdicts. |

serverAppConfigs-docs/LLM/Pipeline Parallelism/llama.cpp RPC.md

@@ -1,9 +0,0 @@
----
-type: reference
-title: "llama.cpp RPC"
-description: "llama.cpp distributed inference via ggml-rpc-server and the --rpc flag: architecture, CLI, versions, limitations, and RPC-compatible speculative decoding."
-tags:
-  - pipeline-parallelism
-  - research
-timestamp: 2026-08-14
----

serverAppConfigs-docs/LLM/Pipeline Parallelism/vLLM Pipeline Parallelism.md

@@ -1,9 +0,0 @@
----
-type: reference
-title: "vLLM Pipeline Parallelism"
-description: "vLLM multi-node pipeline parallelism: flags, Ray/mp backends, interconnect requirements, PP bubble, and PP + speculative decoding support."
-tags:
-  - pipeline-parallelism
-  - research
-timestamp: 2026-08-14
----