Files

T

Frank a5fe045b58 docs(vault): one fact, one owner — repair the doubled pages, single-source the facts

Structural repair:
- 13 pages were the same document twice (each exactly 2x its HEAD size),
  a second frontmatter block whose closing fence had been glued onto a body
  line by a frontmatter rewrite that re-read its own output. Un-doubled by
  cutting at the second frontmatter and keeping the copy whose body is a
  superset; the orphan census reports 0 lost tokens against HEAD.
- log.md: one entry per date, bare `## YYYY-MM-DD` headings, newest first —
  66 dated headings merged into 34, titles demoted to bold leads, every
  content line verified present. The file had been descending at the head
  and ascending at the tail.

Ownership moves:
- New `LLM/frank-medium.md` is the locator for the retired Dankserver
  endpoint: era windows quoted from each era page's own banner, no numbers
  restated, and the two unresolved date conflicts recorded instead of
  silently resolved. `frank-archon.md` links to it instead of restating the
  pre-rename window.
- The four Pipeline-Parallelism stubs (frontmatter only, no body) are gone;
  `Multi-Node Runtimes.md` owns runtime construction, launch, mixed-vendor
  verdicts and cross-split spec decode, and the four links that pointed at
  the stubs were repointed at it or at the drafter page.
- Authentik's three stub pages fold into `Authentik.md`, with the process
  detail and the outpost reference split out so the parent stays under the
  length hint.

Verified corrections (host census + live probe, 2026-10-19):
- The MXFP4 checkpoint two pages called the "on-disk rollback" is gone:
  /home/dank/models/ holds StillDeadcode, rad and tpl.jinja. Stated once, on
  the era page that owns that payload; the swap page now links to it.
- frank-medium-radiance-mxfp4: the 4.24/7 accept rate belongs to the DFlash2
  arm measured on the FP8 target, not to the MXFP4 era; the 3-family soak run
  census is kept once (155 requests) instead of twice (~120 and 155).
- Fleet shapes re-confirmed against the live endpoints before touching any
  page: 3x200000 fp8 KV, 2x200704, 2x262144.

Index rows are regenerated, not hand-carved. Net -1,849 vault lines.

2026-10-19 05:00:43 -07:00

8.8 KiB

Raw Blame History

type, title, description, tags, timestamp

type

title

description

Sample Script Set — Two-Node Qwen3.6-27B Q8 Split

WARNING. This page is for educational reading only. Do not run these scripts. Do not deploy them. They are a template, not a tested product. The RPC backend is a proof of concept that the upstream README calls "fragile and insecure". Running it on a production network is unsafe.

Goal

Run a Qwen3.6-27B Q8 model (about 29 to 32 GB of weights) across two machines using llama.cpp RPC pipeline parallelism. Node A is an NVIDIA desktop with an RTX 3090. Node B is a Strix Halo NUC (Ryzen AI MAX+ 395, Radeon 8060S). Use the largest context that fits on both nodes together.

How the Split Works

graph LR
    subgraph NodeA[NVIDIA Desktop — HEAD]
        A1[llama-server<br/>--model 27B-Q8.gguf<br/>--rpc nodeB:50052]
    end
    subgraph NodeB[Strix Halo NUC — WORKER]
        B1[ggml-rpc-server<br/>--host nodeB --port 50052<br/>--device CUDA0? no — ROCm]
    end
    A1 <-->|hidden states over ethernet| B1

Node A runs the head llama-server. Node B runs ggml-rpc-server, which exposes its GPU as a remote device. The head splits weights and KV cache across its own GPU and the remote device in proportion to free memory. Each token generation passes the hidden state once across the boundary.

Key facts that shape the scripts:

Both nodes must run the exact same llama.cpp build tag. Version skew hangs the handshake.
The head build needs its backend plus RPC. The worker build needs its backend plus RPC. CUDA and ROCm do not mix inside one binary; each node uses its own backend.
Speculative decoding runs on the head. The draft model loads on the head.
The model file must be present on every node with the same bytes.

File Layout

pipeline-pp/
  README.md
  build.sh                  # build llama.cpp with RPC on each node
  workers/worker-strix.sh   # run ggml-rpc-server on the Strix Halo NUC
  head/head-nvidia.sh       # run llama-server on the NVIDIA desktop
  head/frank-27b-pp.env     # tunables for the head
  head/verify.sh            # health and split check

Build Script

File: build.sh

#!/usr/bin/env bash
# Build llama.cpp with RPC support. Run ONE build per node, with that
# node's own backend enabled. Educational only. Do not run in production.
set -euo pipefail
LLAMA_DIR="$1"          # path to a llama.cpp checkout
NODE="$2"               # "nvidia" or "strix"

cmake -S "$LLAMA_DIR" -B "$LLAMA_DIR/build-pp" \
    -DCMAKE_BUILD_TYPE=Release \
    -DGGML_RPC=ON \
    $( [ "$NODE" = nvidia ] && echo -DGGML_CUDA=ON \
      || echo -DGGML_HIP=ON -DGGML_HIP_PLATFORM=amd ) \
    -DGGML_CUDA_NO_GGVK=ON 2>/dev/null || true
cmake --build "$LLAMA_DIR/build-pp" --config Release -j
# Expected binaries: build-pp/bin/ggml-rpc-server, llama-server, llama-cli

Notes: the HEAD does not need a GPU if you offload everything to RPC workers. Keep the head on the NVIDIA desktop so it can also use its local 3090 for part of the split.

Worker Script (Node B, Strix Halo NUC)

File: workers/worker-strix.sh

#!/usr/bin/env bash
# Expose the Strix Halo GPU as an RPC device for a remote head.
# Educational only. Do not run on an unprotected network.
set -euo pipefail
BIN=/home/dank/llama.cpp/build-pp/bin
# Bind to the LAN address. The default binds 127.0.0.1 only.
HOST=192.168.0.143     # the Strix Halo NUC address
PORT=50052
# -c enables a local model-file cache so the split load runs once, not per boot.
exec "$BIN/ggml-rpc-server" --host "$HOST" --port "$PORT" --cache
# Use --device to name the backend device explicitly if auto-detect fails:
#   ggml-rpc-server --host $HOST --port $PORT --device gfx1151

Security note: ggml-rpc-server exposes all memory to any client that can reach the port. Keep it on a trusted LAN. Do not bind it to the public internet.

Head Tunables (Node A, NVIDIA Desktop)

File: head/frank-27b-pp.env

# Educational tunables for the two-node split. Do not deploy.
MODEL_DIR=/home/dank/models
MODEL_FILE=Qwen3.6-27B-Uncensored-HauhauCS-Aggressive-Q8_K_P.gguf  # ~32 GB
# MUCH smaller: use Q8_0 (29.12 GB) if the split must fit more KV.
# Smaller still: Q6_K ~24 GB leaves several GB more room for KV.
MODEL="$MODEL_DIR/$MODEL_FILE"

# Worker (Strix Halo NUC). Multiple --rpc entries allowed; one here.
RPC_SERVERS=192.168.0.143:50052

# Context. 262144 is the native max for this model. Drop to 200000 if
# memory pressure is too high. This is the single-slot ceiling.
CTX_SIZE=262144
PARALLEL=1

# KV cache on GPU. q8_0 K / f16 V saves memory over f16/f16.
CACHE_TYPE_K=q8_0
CACHE_TYPE_V=f16

# Generation batching. Decode is latency-bound over RPC; keep batch modest.
BATCH_SIZE=1024
UBATCH_SIZE=512

# Compute. Match the physical core count on the head box.
THREADS=7

# Sampling (HauhauCS thinking-general defaults).
TEMPERATURE=1.0
TOP_K=20
TOP_P=0.95
MIN_P=0.0

Head Script (Node A, NVIDIA Desktop)

File: head/head-nvidia.sh

#!/usr/bin/env bash
# Run the split model head on the NVIDIA desktop.
# Educational only. Do not run in production.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck disable=SC1091
source "$SCRIPT_DIR/frank-27b-pp.env"

BIN=/home/dank/llama.cpp/build-pp/bin
PORT=7770

RPC_ARGS=()
for s in $RPC_SERVERS; do RPC_ARGS+=(--rpc "$s"); done

# Pick ONE spec variant by uncommenting the matching pair below.

# --- Variant 1: MTP (recommended by research) ---------------------------
# SPEC="--spec-type draft-mtp --spec-draft-model $MODEL_DIR/Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf --spec-draft-n-max 3"

# --- Variant 2: DFlash (block-diffusion draft) ---------------------------
# SPEC="--spec-type draft-dflash --spec-draft-model $MODEL_DIR/Qwen3.6-27B-DFlash-Q8_0.gguf --spec-draft-n-max 8"

# --- Variant 3: ngram-mod (built-in, no draft file) ----------------------
# SPEC="--spec-type ngram-mod --spec-ngram-mod-n-min 48 --spec-ngram-mod-n-max 64 --spec-ngram-mod-n-match 24"

# --- Variant 4: no speculation --------------------------------------------
SPEC="--spec-type none"

exec "$BIN/llama-server" \
    --model "$MODEL" \
    "${RPC_ARGS[@]}" \
    ${SPEC} \
    --ctx-size "$CTX_SIZE" \
    --parallel "$PARALLEL" \
    --cache-type-k "$CACHE_TYPE_K" \
    --cache-type-v "$CACHE_TYPE_V" \
    --batch-size "$BATCH_SIZE" \
    --ubatch-size "$UBATCH_SIZE" \
    --threads "$THREADS" \
    --host 0.0.0.0 --port "$PORT" \
    --alias frank-27b-pp

Spec Variant Summary

Variant	draft model file	--spec-type	Notes
MTP	`Qwen3.6-27B-MTP-ONLY-Q4_K_M.gguf` (~3B)	`draft-mtp`	Parallel-safe, best in benchmarks
DFlash	`Qwen3.6-27B-DFlash-Q8_0.gguf`	`draft-dflash`	Known premature-EOS bug
ngram-mod	none (built-in)	`ngram-mod`	Zero VRAM cost
DSpark	vLLM only	(not in llama.cpp)	Use vLLM PP instead; see below

DSpark does not exist in llama.cpp. It is a vLLM sparse-attention draft. If you want DSpark, use vLLM pipeline parallelism, which has its own limits (no spec decode with PP as of 0.15.x). The llama.cpp path here covers MTP, DFlash, and ngram-mod.

Verify (Node A)

File: head/verify.sh

#!/usr/bin/env bash
# Check the split model is up and that both nodes are in the split.
set -euo pipefail
PORT=7770
# Health must return "ok" after model load (minutes, not seconds).
curl -sf "http://127.0.0.1:$PORT/health"
echo
# Show the device split from the server log.
curl -sf "http://127.0.0.1:$PORT/props" | head -40
# Expected: the model and KV cache distributed across CUDA0 (node A)
# and the remote RPC device (node B).

Context Sizing Math

The model is about 29 to 32 GB. The two machines give about 51 GiB of usable memory. That leaves about 20 to 22 GiB for KV cache. The Qwen3.6-27B KV cache costs about 85 to 100 KiB per token at q8_0 K / f16 V. The budget therefore gives a pool of roughly 200k to 260k tokens. A single 262144-token slot is the practical ceiling for this split.

To run more context, shrink the model quantization. Q6_K (about 24 GB) frees several GB for KV. Do not try to run the 96 GiB driver figure on the Strix Halo as real memory. It is an overcommit ceiling; the physical RAM is 30 GiB.

What This Does Not Cover

The remote draft-model-over-RPC path is not confirmed. The draft stays on the head.
--tensor-split is available to force a manual layer split. The sample uses the automatic proportional split.
RDMA (RoCEv2) transport exists but needs a compatible NIC. It adds a few percent over TCP.

8.8 KiB Raw Blame History