docs: rename docs/ -> serverAppConfigs-docs/ per system-wide PROJECTNAME-docs convention

All in-repo path references (AGENTS.md, README, old scripts, report artifacts, test code paths) rewritten in the same commit; focused-test map extended for .gitignore, old/scripts/*, dankclaw/comfyui/*, and report/* per this repo's own convention ('add one before committing'). Obsidian wikilinks are title-based and unaffected.
2026-08-28 02:20:06 -07:00
parent 6d4a96a423
commit 5829a0bfe8
140 changed files with 1820 additions and 86 deletions
@@ -0,0 +1,261 @@
+---
+type: reference
+title: Sample Script Set
+description: An educational two-node llama.cpp RPC split of Qwen3.6-27B-Q8_K_P over ethernet. For reading only. Do not run or deploy.
+tags:
+  - pipeline-parallelism
+  - sample
+  - llamacpp
+  - educational
+timestamp: 2026-08-14
+---
+
+# Sample Script Set — Two-Node Qwen3.6-27B Q8 Split
+
+> WARNING. This page is for educational reading only. Do not run these
+> scripts. Do not deploy them. They are a template, not a tested product.
+> The RPC backend is a proof of concept that the upstream README calls
+> "fragile and insecure". Running it on a production network is unsafe.
+
+## Goal
+
+Run a Qwen3.6-27B Q8 model (about 29 to 32 GB of weights) across two
+machines using llama.cpp RPC pipeline parallelism. Node A is an NVIDIA
+desktop with an RTX 3090. Node B is a Strix Halo NUC (Ryzen AI MAX+ 395,
+Radeon 8060S). Use the largest context that fits on both nodes together.
+
+## How the Split Works
+
+```mermaid
+graph LR
+    subgraph NodeA[NVIDIA Desktop — HEAD]
+        A1[llama-server<br/>--model 27B-Q8.gguf<br/>--rpc nodeB:50052]
+    end
+    subgraph NodeB[Strix Halo NUC — WORKER]
+        B1[ggml-rpc-server<br/>--host nodeB --port 50052<br/>--device CUDA0? no — ROCm]
+    end
+    A1 <-->|hidden states over ethernet| B1
+```
+
+Node A runs the head `llama-server`. Node B runs `ggml-rpc-server`, which
+exposes its GPU as a remote device. The head splits weights and KV cache
+across its own GPU and the remote device in proportion to free memory.
+Each token generation passes the hidden state once across the boundary.
+
+Key facts that shape the scripts:
+
+- Both nodes must run the exact same llama.cpp build tag. Version skew
+  hangs the handshake.
+- The head build needs its backend plus RPC. The worker build needs its
+  backend plus RPC. CUDA and ROCm do not mix inside one binary; each node
+  uses its own backend.
+- Speculative decoding runs on the head. The draft model loads on the head.
+- The model file must be present on every node with the same bytes.
+
+## File Layout
+
+```
+pipeline-pp/
+  README.md
+  build.sh                  # build llama.cpp with RPC on each node
+  workers/worker-strix.sh   # run ggml-rpc-server on the Strix Halo NUC
+  head/head-nvidia.sh       # run llama-server on the NVIDIA desktop
+  head/frank-27b-pp.env     # tunables for the head
+  head/verify.sh            # health and split check
+```
+
+## Build Script
+
+File: `build.sh`
+
+```bash
+#!/usr/bin/env bash
+# Build llama.cpp with RPC support. Run ONE build per node, with that
+# node's own backend enabled. Educational only. Do not run in production.
+set -euo pipefail
+LLAMA_DIR="$1"          # path to a llama.cpp checkout
+NODE="$2"               # "nvidia" or "strix"
+
+cmake -S "$LLAMA_DIR" -B "$LLAMA_DIR/build-pp" \
+    -DCMAKE_BUILD_TYPE=Release \
+    -DGGML_RPC=ON \
+    $( [ "$NODE" = nvidia ] && echo -DGGML_CUDA=ON \
+      || echo -DGGML_HIP=ON -DGGML_HIP_PLATFORM=amd ) \
+    -DGGML_CUDA_NO_GGVK=ON 2>/dev/null || true
+cmake --build "$LLAMA_DIR/build-pp" --config Release -j
+# Expected binaries: build-pp/bin/ggml-rpc-server, llama-server, llama-cli
+```
+
+Notes: the HEAD does not need a GPU if you offload everything to RPC
+workers. Keep the head on the NVIDIA desktop so it can also use its local
+3090 for part of the split.
+
+## Worker Script (Node B, Strix Halo NUC)
+
+File: `workers/worker-strix.sh`
+
+```bash
+#!/usr/bin/env bash
+# Expose the Strix Halo GPU as an RPC device for a remote head.
+# Educational only. Do not run on an unprotected network.
+set -euo pipefail
+BIN=/home/dank/llama.cpp/build-pp/bin
+# Bind to the LAN address. The default binds 127.0.0.1 only.
+HOST=192.168.0.143     # the Strix Halo NUC address
+PORT=50052
+# -c enables a local model-file cache so the split load runs once, not per boot.
+exec "$BIN/ggml-rpc-server" --host "$HOST" --port "$PORT" --cache
+# Use --device to name the backend device explicitly if auto-detect fails:
+#   ggml-rpc-server --host $HOST --port $PORT --device gfx1151
+```
+
+Security note: `ggml-rpc-server` exposes all memory to any client that can
+reach the port. Keep it on a trusted LAN. Do not bind it to the public
+internet.
+
+## Head Tunables (Node A, NVIDIA Desktop)
+
+File: `head/frank-27b-pp.env`
+
+```env
+# Educational tunables for the two-node split. Do not deploy.
+MODEL_DIR=/home/dank/models
+MODEL_FILE=Qwen3.6-27B-Uncensored-HauhauCS-Aggressive-Q8_K_P.gguf  # ~32 GB
+# MUCH smaller: use Q8_0 (29.12 GB) if the split must fit more KV.
+# Smaller still: Q6_K ~24 GB leaves several GB more room for KV.
+MODEL="$MODEL_DIR/$MODEL_FILE"
+
+# Worker (Strix Halo NUC). Multiple --rpc entries allowed; one here.
+RPC_SERVERS=192.168.0.143:50052
+
+# Context. 262144 is the native max for this model. Drop to 200000 if
+# memory pressure is too high. This is the single-slot ceiling.
+CTX_SIZE=262144
+PARALLEL=1
+
+# KV cache on GPU. q8_0 K / f16 V saves memory over f16/f16.
+CACHE_TYPE_K=q8_0
+CACHE_TYPE_V=f16
+
+# Generation batching. Decode is latency-bound over RPC; keep batch modest.
+BATCH_SIZE=1024
+UBATCH_SIZE=512
+
+# Compute. Match the physical core count on the head box.
+THREADS=7
+
+# Sampling (HauhauCS thinking-general defaults).
+TEMPERATURE=1.0
+TOP_K=20
+TOP_P=0.95
+MIN_P=0.0
+```
+
+## Head Script (Node A, NVIDIA Desktop)
+
+File: `head/head-nvidia.sh`
+
+```bash
+#!/usr/bin/env bash
+# Run the split model head on the NVIDIA desktop.
+# Educational only. Do not run in production.
+set -euo pipefail
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck disable=SC1091
+source "$SCRIPT_DIR/frank-27b-pp.env"
+
+BIN=/home/dank/llama.cpp/build-pp/bin
+PORT=7770
+
+RPC_ARGS=()
+for s in $RPC_SERVERS; do RPC_ARGS+=(--rpc "$s"); done
+
+# Pick ONE spec variant by uncommenting the matching pair below.
+
+# --- Variant 1: MTP (recommended by research) ---------------------------
+# SPEC="--spec-type draft-mtp --spec-draft-model $MODEL_DIR/Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf --spec-draft-n-max 3"
+
+# --- Variant 2: DFlash (block-diffusion draft) ---------------------------
+# SPEC="--spec-type draft-dflash --spec-draft-model $MODEL_DIR/Qwen3.6-27B-DFlash-Q8_0.gguf --spec-draft-n-max 8"
+
+# --- Variant 3: ngram-mod (built-in, no draft file) ----------------------
+# SPEC="--spec-type ngram-mod --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64 --spec-ngram-mod-n-match 24"
+
+# --- Variant 4: no speculation --------------------------------------------
+SPEC="--spec-type none"
+
+exec "$BIN/llama-server" \
+    --model "$MODEL" \
+    "${RPC_ARGS[@]}" \
+    ${SPEC} \
+    --ctx-size "$CTX_SIZE" \
+    --parallel "$PARALLEL" \
+    --cache-type-k "$CACHE_TYPE_K" \
+    --cache-type-v "$CACHE_TYPE_V" \
+    --batch-size "$BATCH_SIZE" \
+    --ubatch-size "$UBATCH_SIZE" \
+    --threads "$THREADS" \
+    --host 0.0.0.0 --port "$PORT" \
+    --alias frank-27b-pp
+```
+
+### Spec Variant Summary
+
+| Variant | draft model file | --spec-type | Notes |
+| ------- | ---------------- | ----------- | ----- |
+| MTP | `Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf` (~3B) | `draft-mtp` | Parallel-safe, best in benchmarks |
+| DFlash | `Qwen3.6-27B-DFlash-Q8_0.gguf` | `draft-dflash` | Known premature-EOS bug |
+| ngram-mod | none (built-in) | `ngram-mod` | Zero VRAM cost |
+| DSpark | vLLM only | (not in llama.cpp) | Use vLLM PP instead; see below |
+
+DSpark does not exist in llama.cpp. It is a vLLM sparse-attention draft. If
+you want DSpark, use vLLM pipeline parallelism, which has its own limits
+(no spec decode with PP as of 0.15.x). The llama.cpp path here covers
+MTP, DFlash, and ngram-mod.
+
+## Verify (Node A)
+
+File: `head/verify.sh`
+
+```bash
+#!/usr/bin/env bash
+# Check the split model is up and that both nodes are in the split.
+set -euo pipefail
+PORT=7770
+# Health must return "ok" after model load (minutes, not seconds).
+curl -sf "http://127.0.0.1:$PORT/health"
+echo
+# Show the device split from the server log.
+curl -sf "http://127.0.0.1:$PORT/props" | head -40
+# Expected: the model and KV cache distributed across CUDA0 (node A)
+# and the remote RPC device (node B).
+```
+
+## Context Sizing Math
+
+The model is about 29 to 32 GB. The two machines give about 51 GiB of
+usable memory. That leaves about 20 to 22 GiB for KV cache. The
+Qwen3.6-27B KV cache costs about 85 to 100 KiB per token at q8_0 K /
+f16 V. The budget therefore gives a pool of roughly 200k to 260k tokens.
+A single 262144-token slot is the practical ceiling for this split.
+
+To run more context, shrink the model quantization. Q6_K (about 24 GB)
+frees several GB for KV. Do not try to run the 96 GiB driver figure on the
+Strix Halo as real memory. It is an overcommit ceiling; the physical RAM
+is 30 GiB.
+
+## What This Does Not Cover
+
+- The remote draft-model-over-RPC path is not confirmed. The draft stays
+  on the head.
+- `--tensor-split` is available to force a manual layer split. The sample
+  uses the automatic proportional split.
+- RDMA (RoCEv2) transport exists but needs a compatible NIC. It adds a few
+  percent over TCP.
+
+## See Also
+
+- [[llama.cpp RPC]]
+- [[Expected Performance]]
+- [[Ethernet Bandwidth Math]]
+- [[Qwen3.6 27B Spec Methods]]