serverAppConfigs/docs/Pipeline Parallelism/Sample Script Set.md

---
type: reference
title: Sample Script Set
description: An educational two-node llama.cpp RPC split of Qwen3.6-27B-Q8_K_P over ethernet. For reading only. Do not run or deploy.
tags:
  - pipeline-parallelism
  - sample
  - llamacpp
  - educational
timestamp: 2026-08-14
---

# Sample Script Set — Two-Node Qwen3.6-27B Q8 Split

> WARNING. This page is for educational reading only. Do not run these
> scripts. Do not deploy them. They are a template, not a tested product.
> The RPC backend is a proof of concept that the upstream README calls
> "fragile and insecure". Running it on a production network is unsafe.

## Goal

Run a Qwen3.6-27B Q8 model (about 29 to 32 GB of weights) across two
machines using llama.cpp RPC pipeline parallelism. Node A is an NVIDIA
desktop with an RTX 3090. Node B is a Strix Halo NUC (Ryzen AI MAX+ 395,
Radeon 8060S). Use the largest context that fits on both nodes together.

## How the Split Works

```mermaid
graph LR
    subgraph NodeA[NVIDIA Desktop — HEAD]
        A1[llama-server<br/>--model 27B-Q8.gguf<br/>--rpc nodeB:50052]
    end
    subgraph NodeB[Strix Halo NUC — WORKER]
        B1[ggml-rpc-server<br/>--host nodeB --port 50052<br/>--device CUDA0? no — ROCm]
    end
    A1 <-->|hidden states over ethernet| B1
```

Node A runs the head `llama-server`. Node B runs `ggml-rpc-server`, which
exposes its GPU as a remote device. The head splits weights and KV cache
across its own GPU and the remote device in proportion to free memory.
Each token generation passes the hidden state once across the boundary.

Key facts that shape the scripts:

- Both nodes must run the exact same llama.cpp build tag. Version skew
  hangs the handshake.
- The head build needs its backend plus RPC. The worker build needs its
  backend plus RPC. CUDA and ROCm do not mix inside one binary; each node
  uses its own backend.
- Speculative decoding runs on the head. The draft model loads on the head.
- The model file must be present on every node with the same bytes.

## File Layout

```
pipeline-pp/
  README.md
  build.sh                  # build llama.cpp with RPC on each node
  workers/worker-strix.sh   # run ggml-rpc-server on the Strix Halo NUC
  head/head-nvidia.sh       # run llama-server on the NVIDIA desktop
  head/frank-27b-pp.env     # tunables for the head
  head/verify.sh            # health and split check
```

## Build Script

File: `build.sh`

```bash
#!/usr/bin/env bash
# Build llama.cpp with RPC support. Run ONE build per node, with that
# node's own backend enabled. Educational only. Do not run in production.
set -euo pipefail
LLAMA_DIR="$1"          # path to a llama.cpp checkout
NODE="$2"               # "nvidia" or "strix"

cmake -S "$LLAMA_DIR" -B "$LLAMA_DIR/build-pp" \
    -DCMAKE_BUILD_TYPE=Release \
    -DGGML_RPC=ON \
    $( [ "$NODE" = nvidia ] && echo -DGGML_CUDA=ON \
      || echo -DGGML_HIP=ON -DGGML_HIP_PLATFORM=amd ) \
    -DGGML_CUDA_NO_GGVK=ON 2>/dev/null || true
cmake --build "$LLAMA_DIR/build-pp" --config Release -j
# Expected binaries: build-pp/bin/ggml-rpc-server, llama-server, llama-cli
```

Notes: the HEAD does not need a GPU if you offload everything to RPC
workers. Keep the head on the NVIDIA desktop so it can also use its local
3090 for part of the split.

## Worker Script (Node B, Strix Halo NUC)

File: `workers/worker-strix.sh`

```bash
#!/usr/bin/env bash
# Expose the Strix Halo GPU as an RPC device for a remote head.
# Educational only. Do not run on an unprotected network.
set -euo pipefail
BIN=/home/dank/llama.cpp/build-pp/bin
# Bind to the LAN address. The default binds 127.0.0.1 only.
HOST=192.168.0.143     # the Strix Halo NUC address
PORT=50052
# -c enables a local model-file cache so the split load runs once, not per boot.
exec "$BIN/ggml-rpc-server" --host "$HOST" --port "$PORT" --cache
# Use --device to name the backend device explicitly if auto-detect fails:
#   ggml-rpc-server --host $HOST --port $PORT --device gfx1151
```

Security note: `ggml-rpc-server` exposes all memory to any client that can
reach the port. Keep it on a trusted LAN. Do not bind it to the public
internet.

## Head Tunables (Node A, NVIDIA Desktop)

File: `head/frank-27b-pp.env`

```env
# Educational tunables for the two-node split. Do not deploy.
MODEL_DIR=/home/dank/models
MODEL_FILE=Qwen3.6-27B-Uncensored-HauhauCS-Aggressive-Q8_K_P.gguf  # ~32 GB
# MUCH smaller: use Q8_0 (29.12 GB) if the split must fit more KV.
# Smaller still: Q6_K ~24 GB leaves several GB more room for KV.
MODEL="$MODEL_DIR/$MODEL_FILE"

# Worker (Strix Halo NUC). Multiple --rpc entries allowed; one here.
RPC_SERVERS=192.168.0.143:50052

# Context. 262144 is the native max for this model. Drop to 200000 if
# memory pressure is too high. This is the single-slot ceiling.
CTX_SIZE=262144
PARALLEL=1

# KV cache on GPU. q8_0 K / f16 V saves memory over f16/f16.
CACHE_TYPE_K=q8_0
CACHE_TYPE_V=f16

# Generation batching. Decode is latency-bound over RPC; keep batch modest.
BATCH_SIZE=1024
UBATCH_SIZE=512

# Compute. Match the physical core count on the head box.
THREADS=7

# Sampling (HauhauCS thinking-general defaults).
TEMPERATURE=1.0
TOP_K=20
TOP_P=0.95
MIN_P=0.0
```

## Head Script (Node A, NVIDIA Desktop)

File: `head/head-nvidia.sh`

```bash
#!/usr/bin/env bash
# Run the split model head on the NVIDIA desktop.
# Educational only. Do not run in production.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck disable=SC1091
source "$SCRIPT_DIR/frank-27b-pp.env"

BIN=/home/dank/llama.cpp/build-pp/bin
PORT=7770

RPC_ARGS=()
for s in $RPC_SERVERS; do RPC_ARGS+=(--rpc "$s"); done

# Pick ONE spec variant by uncommenting the matching pair below.

# --- Variant 1: MTP (recommended by research) ---------------------------
# SPEC="--spec-type draft-mtp --spec-draft-model $MODEL_DIR/Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf --spec-draft-n-max 3"

# --- Variant 2: DFlash (block-diffusion draft) ---------------------------
# SPEC="--spec-type draft-dflash --spec-draft-model $MODEL_DIR/Qwen3.6-27B-DFlash-Q8_0.gguf --spec-draft-n-max 8"

# --- Variant 3: ngram-mod (built-in, no draft file) ----------------------
# SPEC="--spec-type ngram-mod --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64 --spec-ngram-mod-n-match 24"

# --- Variant 4: no speculation --------------------------------------------
SPEC="--spec-type none"

exec "$BIN/llama-server" \
    --model "$MODEL" \
    "${RPC_ARGS[@]}" \
    ${SPEC} \
    --ctx-size "$CTX_SIZE" \
    --parallel "$PARALLEL" \
    --cache-type-k "$CACHE_TYPE_K" \
    --cache-type-v "$CACHE_TYPE_V" \
    --batch-size "$BATCH_SIZE" \
    --ubatch-size "$UBATCH_SIZE" \
    --threads "$THREADS" \
    --host 0.0.0.0 --port "$PORT" \
    --alias frank-27b-pp
```

### Spec Variant Summary

| Variant | draft model file | --spec-type | Notes |
| ------- | ---------------- | ----------- | ----- |
| MTP | `Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf` (~3B) | `draft-mtp` | Parallel-safe, best in benchmarks |
| DFlash | `Qwen3.6-27B-DFlash-Q8_0.gguf` | `draft-dflash` | Known premature-EOS bug |
| ngram-mod | none (built-in) | `ngram-mod` | Zero VRAM cost |
| DSpark | vLLM only | (not in llama.cpp) | Use vLLM PP instead; see below |

DSpark does not exist in llama.cpp. It is a vLLM sparse-attention draft. If
you want DSpark, use vLLM pipeline parallelism, which has its own limits
(no spec decode with PP as of 0.15.x). The llama.cpp path here covers
MTP, DFlash, and ngram-mod.

## Verify (Node A)

File: `head/verify.sh`

```bash
#!/usr/bin/env bash
# Check the split model is up and that both nodes are in the split.
set -euo pipefail
PORT=7770
# Health must return "ok" after model load (minutes, not seconds).
curl -sf "http://127.0.0.1:$PORT/health"
echo
# Show the device split from the server log.
curl -sf "http://127.0.0.1:$PORT/props" | head -40
# Expected: the model and KV cache distributed across CUDA0 (node A)
# and the remote RPC device (node B).
```

## Context Sizing Math

The model is about 29 to 32 GB. The two machines give about 51 GiB of
usable memory. That leaves about 20 to 22 GiB for KV cache. The
Qwen3.6-27B KV cache costs about 85 to 100 KiB per token at q8_0 K /
f16 V. The budget therefore gives a pool of roughly 200k to 260k tokens.
A single 262144-token slot is the practical ceiling for this split.

To run more context, shrink the model quantization. Q6_K (about 24 GB)
frees several GB for KV. Do not try to run the 96 GiB driver figure on the
Strix Halo as real memory. It is an overcommit ceiling; the physical RAM
is 30 GiB.

## What This Does Not Cover

- The remote draft-model-over-RPC path is not confirmed. The draft stays
  on the head.
- `--tensor-split` is available to force a manual layer split. The sample
  uses the automatic proportional split.
- RDMA (RoCEv2) transport exists but needs a compatible NIC. It adds a few
  percent over TCP.

## See Also

- [[llama.cpp RPC]]
- [[Expected Performance]]
- [[Ethernet Bandwidth Math]]
- [[Qwen3.6 27B Spec Methods]]