serverAppConfigs/docs/Pipeline Parallelism/Sample Script Set.md

type, title, description, tags, timestamp

type	title	description	tags	timestamp
reference	Sample Script Set	An educational two-node llama.cpp RPC split of Qwen3.6-27B-Q8_K_P over ethernet. For reading only. Do not run or deploy.	pipeline-parallelism sample llamacpp educational	2026-08-14

Sample Script Set — Two-Node Qwen3.6-27B Q8 Split

WARNING. This page is for educational reading only. Do not run these scripts. Do not deploy them. They are a template, not a tested product. The RPC backend is a proof of concept that the upstream README calls "fragile and insecure". Running it on a production network is unsafe.

Goal

Run a Qwen3.6-27B Q8 model (about 29 to 32 GB of weights) across two machines using llama.cpp RPC pipeline parallelism. Node A is an NVIDIA desktop with an RTX 3090. Node B is a Strix Halo NUC (Ryzen AI MAX+ 395, Radeon 8060S). Use the largest context that fits on both nodes together.

How the Split Works

graph LR
    subgraph NodeA[NVIDIA Desktop — HEAD]
        A1[llama-server<br/>--model 27B-Q8.gguf<br/>--rpc nodeB:50052]
    end
    subgraph NodeB[Strix Halo NUC — WORKER]
        B1[ggml-rpc-server<br/>--host nodeB --port 50052<br/>--device CUDA0? no — ROCm]
    end
    A1 <-->|hidden states over ethernet| B1

Node A runs the head llama-server. Node B runs ggml-rpc-server, which exposes its GPU as a remote device. The head splits weights and KV cache across its own GPU and the remote device in proportion to free memory. Each token generation passes the hidden state once across the boundary.

Key facts that shape the scripts:

• Both nodes must run the exact same llama.cpp build tag. Version skew hangs the handshake.
• The head build needs its backend plus RPC. The worker build needs its backend plus RPC. CUDA and ROCm do not mix inside one binary; each node uses its own backend.
• Speculative decoding runs on the head. The draft model loads on the head.
• The model file must be present on every node with the same bytes.

File Layout

pipeline-pp/
  README.md
  build.sh                  # build llama.cpp with RPC on each node
  workers/worker-strix.sh   # run ggml-rpc-server on the Strix Halo NUC
  head/head-nvidia.sh       # run llama-server on the NVIDIA desktop
  head/frank-27b-pp.env     # tunables for the head
  head/verify.sh            # health and split check

Build Script

File: build.sh

#!/usr/bin/env bash
# Build llama.cpp with RPC support. Run ONE build per node, with that
# node's own backend enabled. Educational only. Do not run in production.
set -euo pipefail
LLAMA_DIR="$1"          # path to a llama.cpp checkout
NODE="$2"               # "nvidia" or "strix"

cmake -S "$LLAMA_DIR" -B "$LLAMA_DIR/build-pp" \
    -DCMAKE_BUILD_TYPE=Release \
    -DGGML_RPC=ON \
    $( [ "$NODE" = nvidia ] && echo -DGGML_CUDA=ON \
      || echo -DGGML_HIP=ON -DGGML_HIP_PLATFORM=amd ) \
    -DGGML_CUDA_NO_GGVK=ON 2>/dev/null || true
cmake --build "$LLAMA_DIR/build-pp" --config Release -j
# Expected binaries: build-pp/bin/ggml-rpc-server, llama-server, llama-cli

Notes: the HEAD does not need a GPU if you offload everything to RPC workers. Keep the head on the NVIDIA desktop so it can also use its local 3090 for part of the split.

Worker Script (Node B, Strix Halo NUC)

File: workers/worker-strix.sh

#!/usr/bin/env bash
# Expose the Strix Halo GPU as an RPC device for a remote head.
# Educational only. Do not run on an unprotected network.
set -euo pipefail
BIN=/home/dank/llama.cpp/build-pp/bin
# Bind to the LAN address. The default binds 127.0.0.1 only.
HOST=192.168.0.143     # the Strix Halo NUC address
PORT=50052
# -c enables a local model-file cache so the split load runs once, not per boot.
exec "$BIN/ggml-rpc-server" --host "$HOST" --port "$PORT" --cache
# Use --device to name the backend device explicitly if auto-detect fails:
#   ggml-rpc-server --host $HOST --port $PORT --device gfx1151

Security note: ggml-rpc-server exposes all memory to any client that can reach the port. Keep it on a trusted LAN. Do not bind it to the public internet.

Head Tunables (Node A, NVIDIA Desktop)

File: head/frank-27b-pp.env

# Educational tunables for the two-node split. Do not deploy.
MODEL_DIR=/home/dank/models
MODEL_FILE=Qwen3.6-27B-Uncensored-HauhauCS-Aggressive-Q8_K_P.gguf  # ~32 GB
# MUCH smaller: use Q8_0 (29.12 GB) if the split must fit more KV.
# Smaller still: Q6_K ~24 GB leaves several GB more room for KV.
MODEL="$MODEL_DIR/$MODEL_FILE"

# Worker (Strix Halo NUC). Multiple --rpc entries allowed; one here.
RPC_SERVERS=192.168.0.143:50052

# Context. 262144 is the native max for this model. Drop to 200000 if
# memory pressure is too high. This is the single-slot ceiling.
CTX_SIZE=262144
PARALLEL=1

# KV cache on GPU. q8_0 K / f16 V saves memory over f16/f16.
CACHE_TYPE_K=q8_0
CACHE_TYPE_V=f16

# Generation batching. Decode is latency-bound over RPC; keep batch modest.
BATCH_SIZE=1024
UBATCH_SIZE=512

# Compute. Match the physical core count on the head box.
THREADS=7

# Sampling (HauhauCS thinking-general defaults).
TEMPERATURE=1.0
TOP_K=20
TOP_P=0.95
MIN_P=0.0

Head Script (Node A, NVIDIA Desktop)

File: head/head-nvidia.sh

#!/usr/bin/env bash
# Run the split model head on the NVIDIA desktop.
# Educational only. Do not run in production.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck disable=SC1091
source "$SCRIPT_DIR/frank-27b-pp.env"

BIN=/home/dank/llama.cpp/build-pp/bin
PORT=7770

RPC_ARGS=()
for s in $RPC_SERVERS; do RPC_ARGS+=(--rpc "$s"); done

# Pick ONE spec variant by uncommenting the matching pair below.

# --- Variant 1: MTP (recommended by research) ---------------------------
# SPEC="--spec-type draft-mtp --spec-draft-model $MODEL_DIR/Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf --spec-draft-n-max 3"

# --- Variant 2: DFlash (block-diffusion draft) ---------------------------
# SPEC="--spec-type draft-dflash --spec-draft-model $MODEL_DIR/Qwen3.6-27B-DFlash-Q8_0.gguf --spec-draft-n-max 8"

# --- Variant 3: ngram-mod (built-in, no draft file) ----------------------
# SPEC="--spec-type ngram-mod --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64 --spec-ngram-mod-n-match 24"

# --- Variant 4: no speculation --------------------------------------------
SPEC="--spec-type none"

exec "$BIN/llama-server" \
    --model "$MODEL" \
    "${RPC_ARGS[@]}" \
    ${SPEC} \
    --ctx-size "$CTX_SIZE" \
    --parallel "$PARALLEL" \
    --cache-type-k "$CACHE_TYPE_K" \
    --cache-type-v "$CACHE_TYPE_V" \
    --batch-size "$BATCH_SIZE" \
    --ubatch-size "$UBATCH_SIZE" \
    --threads "$THREADS" \
    --host 0.0.0.0 --port "$PORT" \
    --alias frank-27b-pp

Spec Variant Summary

Variant	draft model file	--spec-type	Notes
MTP	Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf (~3B)	draft-mtp	Parallel-safe, best in benchmarks
DFlash	Qwen3.6-27B-DFlash-Q8_0.gguf	draft-dflash	Known premature-EOS bug
ngram-mod	none (built-in)	ngram-mod	Zero VRAM cost
DSpark	vLLM only	(not in llama.cpp)	Use vLLM PP instead; see below

DSpark does not exist in llama.cpp. It is a vLLM sparse-attention draft. If you want DSpark, use vLLM pipeline parallelism, which has its own limits (no spec decode with PP as of 0.15.x). The llama.cpp path here covers MTP, DFlash, and ngram-mod.

Verify (Node A)

File: head/verify.sh

#!/usr/bin/env bash
# Check the split model is up and that both nodes are in the split.
set -euo pipefail
PORT=7770
# Health must return "ok" after model load (minutes, not seconds).
curl -sf "http://127.0.0.1:$PORT/health"
echo
# Show the device split from the server log.
curl -sf "http://127.0.0.1:$PORT/props" | head -40
# Expected: the model and KV cache distributed across CUDA0 (node A)
# and the remote RPC device (node B).

Context Sizing Math

The model is about 29 to 32 GB. The two machines give about 51 GiB of usable memory. That leaves about 20 to 22 GiB for KV cache. The Qwen3.6-27B KV cache costs about 85 to 100 KiB per token at q8_0 K / f16 V. The budget therefore gives a pool of roughly 200k to 260k tokens. A single 262144-token slot is the practical ceiling for this split.

To run more context, shrink the model quantization. Q6_K (about 24 GB) frees several GB for KV. Do not try to run the 96 GiB driver figure on the Strix Halo as real memory. It is an overcommit ceiling; the physical RAM is 30 GiB.

What This Does Not Cover

• The remote draft-model-over-RPC path is not confirmed. The draft stays on the head.
• --tensor-split is available to force a manual layer split. The sample uses the automatic proportional split.
• RDMA (RoCEv2) transport exists but needs a compatible NIC. It adds a few percent over TCP.

See Also

• llama.cpp RPC
• Expected Performance
• Ethernet Bandwidth Math
• Qwen3.6 27B Spec Methods