# Pipeline Parallelism

This section covers running one large model across two machines. It focuses
on pipeline parallelism (layer split) over ethernet. The research answers
what speeds are possible and which tools work. The sample scripts show a
two-node Qwen3.6-27B Q8 split for reading only.

## Terminology

Pipeline parallelism (PP) splits model layers across machines. Each machine
computes its layers and passes hidden states to the next machine. Tensor
parallelism (TP) splits each layer across machines and requires an
all-reduce per layer. PP is the only realistic option over ethernet.

Speculative decoding is a separate feature. It uses a small draft model (or
n-grams) to propose tokens that the main model verifies in one pass. It can
run together with PP in llama.cpp.

## The Two Runtimes

```mermaid
graph LR
    subgraph llamaRPC[llama.cpp RPC]
        A1[ggml-rpc-server on node B] --> A2[llama-server on node A<br/>--rpc nodeB:50052]
    end
    subgraph vllmPP[vLLM PP]
        B1[vllm serve node 0<br/>tp=1 pp=2 rank 0] <-->|hidden states| B2[vllm serve node 1<br/>tp=1 pp=2 rank 1]
    end
```

  heterogeneous nodes over ethernet.
  multiprocessing and does not compose with speculative decoding yet.

## What the Research Found

- [[Ethernet Bandwidth Math]] — Bytes-per-token math for pipeline versus tensor parallelism, and the link budgets that follow from it.
  (about 10 KiB per token for a 27B model). 1 GbE supports hundreds of
  tokens per second on paper; the real limit is latency and RPC overhead.
  works today via llama.cpp RPC with separate CUDA and ROCm builds. The
  exact NVIDIA-desktop + Strix-Halo-NUC pair has not been benchmarked with
  published numbers yet.
  methods (MTP, DFlash, ngram-mod) work with RPC. DSpark is vLLM-only.
- [[Expected Performance]] — Estimated split-model throughput over ethernet, with the benchmark anchors the estimates rest on.
  across an RTX 3090 and a Strix Halo NUC over 1/2.5/10 GbE.

## Deployment

- [[Qwen3.8 PP Deployment]] — The real split deployment: which node is head and which is worker, the flags, and the verified gotchas.
  (Strix), worker on Dankserver (RTX 3090), b10433 on both sides, scripts in
  `scripts/`, morning-of test runbook. Verified gotchas: no `--device` with
  `--rpc`, default split is proportional to reported free memory (pin 1,1),
  the iGPU side defaults to the ROCm backend.

## Sample Scripts

- [[Sample Script Set]] — An educational two-node RPC split script set, for reading only — never run it or deploy it.
  Qwen3.6-27B-Q8_K_P with MTP, DFlash, and ngram-mod variants, sized for
  the largest context that fits. For educational reading only. Do not run
  it in production.

## See Also

- [[DFlash DSpark Speculative Decoding]]
- [[frank-zealot]]
- [[Dankserver System Specs]]
- [[Dankclaw System Specs]]

## Pages

| Page | Owns |
|---|---|
| [[Multi-Node Runtimes]] | How each two-machine runtime is built and launched, what it can combine with, and the mixed-vendor verdicts. |