--- type: reference title: Sample Script Set description: "An educational two-node RPC split script set, for reading only — never run it or deploy it." tags: - pipeline-parallelism - sample - llamacpp - educational timestamp: 2026-08-14 --- # Sample Script Set — Two-Node Qwen3.6-27B Q8 Split > WARNING. This page is for educational reading only. Do not run these > scripts. Do not deploy them. They are a template, not a tested product. > The RPC backend is a proof of concept that the upstream README calls > "fragile and insecure". Running it on a production network is unsafe. ## Goal Run a Qwen3.6-27B Q8 model (about 29 to 32 GB of weights) across two machines using llama.cpp RPC pipeline parallelism. Node A is an NVIDIA desktop with an RTX 3090. Node B is a Strix Halo NUC (Ryzen AI MAX+ 395, Radeon 8060S). Use the largest context that fits on both nodes together. ## How the Split Works ```mermaid graph LR subgraph NodeA[NVIDIA Desktop — HEAD] A1[llama-server
--model 27B-Q8.gguf
--rpc nodeB:50052] end subgraph NodeB[Strix Halo NUC — WORKER] B1[ggml-rpc-server
--host nodeB --port 50052
--device CUDA0? no — ROCm] end A1 <-->|hidden states over ethernet| B1 ``` Node A runs the head `llama-server`. Node B runs `ggml-rpc-server`, which exposes its GPU as a remote device. The head splits weights and KV cache across its own GPU and the remote device in proportion to free memory. Each token generation passes the hidden state once across the boundary. Key facts that shape the scripts: - Both nodes must run the exact same llama.cpp build tag. Version skew hangs the handshake. - The head build needs its backend plus RPC. The worker build needs its backend plus RPC. CUDA and ROCm do not mix inside one binary; each node uses its own backend. - Speculative decoding runs on the head. The draft model loads on the head. - The model file must be present on every node with the same bytes. ## File Layout ``` pipeline-pp/ README.md build.sh # build llama.cpp with RPC on each node workers/worker-strix.sh # run ggml-rpc-server on the Strix Halo NUC head/head-nvidia.sh # run llama-server on the NVIDIA desktop head/frank-27b-pp.env # tunables for the head head/verify.sh # health and split check ``` ## Build Script File: `build.sh` ```bash #!/usr/bin/env bash # Build llama.cpp with RPC support. Run ONE build per node, with that # node's own backend enabled. Educational only. Do not run in production. set -euo pipefail LLAMA_DIR="$1" # path to a llama.cpp checkout NODE="$2" # "nvidia" or "strix" cmake -S "$LLAMA_DIR" -B "$LLAMA_DIR/build-pp" \ -DCMAKE_BUILD_TYPE=Release \ -DGGML_RPC=ON \ $( [ "$NODE" = nvidia ] && echo -DGGML_CUDA=ON \ || echo -DGGML_HIP=ON -DGGML_HIP_PLATFORM=amd ) \ -DGGML_CUDA_NO_GGVK=ON 2>/dev/null || true cmake --build "$LLAMA_DIR/build-pp" --config Release -j # Expected binaries: build-pp/bin/ggml-rpc-server, llama-server, llama-cli ``` Notes: the HEAD does not need a GPU if you offload everything to RPC workers. Keep the head on the NVIDIA desktop so it can also use its local 3090 for part of the split. ## Worker Script (Node B, Strix Halo NUC) File: `workers/worker-strix.sh` ```bash #!/usr/bin/env bash # Expose the Strix Halo GPU as an RPC device for a remote head. # Educational only. Do not run on an unprotected network. set -euo pipefail BIN=/home/dank/llama.cpp/build-pp/bin # Bind to the LAN address. The default binds 127.0.0.1 only. HOST=192.168.0.143 # the Strix Halo NUC address PORT=50052 # -c enables a local model-file cache so the split load runs once, not per boot. exec "$BIN/ggml-rpc-server" --host "$HOST" --port "$PORT" --cache # Use --device to name the backend device explicitly if auto-detect fails: # ggml-rpc-server --host $HOST --port $PORT --device gfx1151 ``` Security note: `ggml-rpc-server` exposes all memory to any client that can reach the port. Keep it on a trusted LAN. Do not bind it to the public internet. ## Head Tunables (Node A, NVIDIA Desktop) File: `head/frank-27b-pp.env` ```env # Educational tunables for the two-node split. Do not deploy. MODEL_DIR=/home/dank/models MODEL_FILE=Qwen3.6-27B-Uncensored-HauhauCS-Aggressive-Q8_K_P.gguf # ~32 GB # MUCH smaller: use Q8_0 (29.12 GB) if the split must fit more KV. # Smaller still: Q6_K ~24 GB leaves several GB more room for KV. MODEL="$MODEL_DIR/$MODEL_FILE" # Worker (Strix Halo NUC). Multiple --rpc entries allowed; one here. RPC_SERVERS=192.168.0.143:50052 # Context. 262144 is the native max for this model. Drop to 200000 if # memory pressure is too high. This is the single-slot ceiling. CTX_SIZE=262144 PARALLEL=1 # KV cache on GPU. q8_0 K / f16 V saves memory over f16/f16. CACHE_TYPE_K=q8_0 CACHE_TYPE_V=f16 # Generation batching. Decode is latency-bound over RPC; keep batch modest. BATCH_SIZE=1024 UBATCH_SIZE=512 # Compute. Match the physical core count on the head box. THREADS=7 # Sampling (HauhauCS thinking-general defaults). TEMPERATURE=1.0 TOP_K=20 TOP_P=0.95 MIN_P=0.0 ``` ## Head Script (Node A, NVIDIA Desktop) File: `head/head-nvidia.sh` ```bash #!/usr/bin/env bash # Run the split model head on the NVIDIA desktop. # Educational only. Do not run in production. set -euo pipefail SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" # shellcheck disable=SC1091 source "$SCRIPT_DIR/frank-27b-pp.env" BIN=/home/dank/llama.cpp/build-pp/bin PORT=7770 RPC_ARGS=() for s in $RPC_SERVERS; do RPC_ARGS+=(--rpc "$s"); done # Pick ONE spec variant by uncommenting the matching pair below. # --- Variant 1: MTP (recommended by research) --------------------------- # SPEC="--spec-type draft-mtp --spec-draft-model $MODEL_DIR/Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf --spec-draft-n-max 3" # --- Variant 2: DFlash (block-diffusion draft) --------------------------- # SPEC="--spec-type draft-dflash --spec-draft-model $MODEL_DIR/Qwen3.6-27B-DFlash-Q8_0.gguf --spec-draft-n-max 8" # --- Variant 3: ngram-mod (built-in, no draft file) ---------------------- # SPEC="--spec-type ngram-mod --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64 --spec-ngram-mod-n-match 24" # --- Variant 4: no speculation -------------------------------------------- SPEC="--spec-type none" exec "$BIN/llama-server" \ --model "$MODEL" \ "${RPC_ARGS[@]}" \ ${SPEC} \ --ctx-size "$CTX_SIZE" \ --parallel "$PARALLEL" \ --cache-type-k "$CACHE_TYPE_K" \ --cache-type-v "$CACHE_TYPE_V" \ --batch-size "$BATCH_SIZE" \ --ubatch-size "$UBATCH_SIZE" \ --threads "$THREADS" \ --host 0.0.0.0 --port "$PORT" \ --alias frank-27b-pp ``` ### Spec Variant Summary | Variant | draft model file | --spec-type | Notes | | ------- | ---------------- | ----------- | ----- | | MTP | `Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf` (~3B) | `draft-mtp` | Parallel-safe, best in benchmarks | | DFlash | `Qwen3.6-27B-DFlash-Q8_0.gguf` | `draft-dflash` | Known premature-EOS bug | | ngram-mod | none (built-in) | `ngram-mod` | Zero VRAM cost | | DSpark | vLLM only | (not in llama.cpp) | Use vLLM PP instead; see below | DSpark does not exist in llama.cpp. It is a vLLM sparse-attention draft. If you want DSpark, use vLLM pipeline parallelism, which has its own limits (no spec decode with PP as of 0.15.x). The llama.cpp path here covers MTP, DFlash, and ngram-mod. ## Verify (Node A) File: `head/verify.sh` ```bash #!/usr/bin/env bash # Check the split model is up and that both nodes are in the split. set -euo pipefail PORT=7770 # Health must return "ok" after model load (minutes, not seconds). curl -sf "http://127.0.0.1:$PORT/health" echo # Show the device split from the server log. curl -sf "http://127.0.0.1:$PORT/props" | head -40 # Expected: the model and KV cache distributed across CUDA0 (node A) # and the remote RPC device (node B). ``` ## Context Sizing Math The model is about 29 to 32 GB. The two machines give about 51 GiB of usable memory. That leaves about 20 to 22 GiB for KV cache. The Qwen3.6-27B KV cache costs about 85 to 100 KiB per token at q8_0 K / f16 V. The budget therefore gives a pool of roughly 200k to 260k tokens. A single 262144-token slot is the practical ceiling for this split. To run more context, shrink the model quantization. Q6_K (about 24 GB) frees several GB for KV. Do not try to run the 96 GiB driver figure on the Strix Halo as real memory. It is an overcommit ceiling; the physical RAM is 30 GiB. ## What This Does Not Cover - The remote draft-model-over-RPC path is not confirmed. The draft stays on the head. - `--tensor-split` is available to force a manual layer split. The sample uses the automatic proportional split. - RDMA (RoCEv2) transport exists but needs a compatible NIC. It adds a few percent over TCP. ## See Also - [[Qwen3.8 PP Deployment]] - [[Expected Performance]] - [[Ethernet Bandwidth Math]] - [[DFlash DSpark Speculative Decoding]]