dank/serverAppConfigs
1
0 Fork 0
Code Issues 4 Pull Requests Actions Packages Projects Releases Wiki Activity

PP kit + Strix research: Qwen3.8-27B RPC split (Dankclaw head / Dankserver worker)

Browse Source 
- scripts: serve-pp-head/worker, envs, systemd units, pp-test.sh orchestrator, pp-bench.py
- head design verified in b10433 source + live 4B smoke test:
  * no --device with --rpc (RPC device silently dropped otherwise)
  * default device selection = [RPC0, ROCm0] (HIP registers before Vulkan)
  * default split proportional to reported free mem = ~20/80 here, pin 1,1
  * -dio required for large model over RPC
- docs: qwen3.8-strix-optimizations (ranked no-quant-downgrade levers,
  quant landscape, 2026-08-16 test matrix), PP deployment runbook,
  index updates, Dankclaw Improvements corrections (Vulkan0, no power_profile sysfs)
- worker built at 9b05354ec in /home/dank/llama-cpp-rpc worktree (CUDA+RPC)
- second head build build-vulkan-rpc (HIP off) for backend A/B
This commit is contained in:
Frank committed 2026-08-16 05:40:27 -07:00
1 parent c775077016
commit 9544c36afc
13 files changed
+1037 -5
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

scripts/frank-pp-head.env
+76
View File
@@ -0,0 +1,76 @@
# Head tunables — Qwen3.8-27B pipeline-parallel head (Dankclaw).
#
# The head is the LOCAL side of the split. It runs llama-server (b10433,
# HIP+Vulkan+RPC build in /home/dank/llama.cpp/build) with --rpc pointing at
# the worker on Dankserver. The model file lives on this machine only; the
# worker receives its tensor share over the network at load time.
#
# CRITICAL: this script must NOT pass --device. When --device is set,
# llama.cpp uses exactly that device list and silently drops the RPC device
# (src/llama.cpp llama_prepare_model_devices). Default device selection
# inserts RPC servers at the FRONT of the device list, so the split becomes
# [RPC0, iGPU] and --tensor-split index 0 is the worker.
#
# WARNING: do not name any variable here TEMP, HOST, or GPU_MAX_HEAP_SIZE
# (HIP runtime collision, see the 2026-08-13 TEMP incident).
MODEL_DIR=/home/dank/models
MODEL_FILE=Qwen3.8-27B-UD-Q8_K_XL.gguf
HEAD_BIN=/home/dank/llama.cpp/build/bin/llama-server
ALIAS=frank-pp-large
BIND_ADDR=0.0.0.0
LISTEN_PORT=7770
# Comma-separated RPC endpoints (host:port). One worker here.
RPC_SERVERS=192.168.0.235:50052
# Model split fractions, one per device in order [RPC0, iGPU].
# 1,1 = 50/50: the 3090 gets the first ~32 layers, the Strix the rest.
# Leave empty for the default proportional-to-free-memory split; note the
# Strix reports ~96 GiB (GTT ceiling) vs the 3090's ~24 GB, so the default
# lands around 20% of layers on the 3090 and 80% on the Strix.
TENSOR_SPLIT=1,1
# Context pool. 131072 single slot keeps the KV small (~4.5 GB total at
# q8_0) on both sides; raise for high-context tests (see docs).
CTX_SIZE=131072
PARALLEL=1
# Speculative decoding. Matches production frank-default-large: MTP (bundled
# head, n_max 4 per the published tuning curve) with the ngram-mod fallback.
# Set SPEC_NONE=1 for a raw baseline with no speculation.
SPEC_NONE=0
SPEC_DEFAULT=1
MTP=1
SPEC_DRAFT_N_MAX=4
# Optional standalone MTP draft head (e.g. the Q4_0 ggml-org file). When set,
# it replaces the bundled in-model head via --spec-draft-model.
SPEC_DRAFT_MODEL=
SPEC_NGRAM_MOD_N_MIN=48
SPEC_NGRAM_MOD_N_MAX=64
SPEC_NGRAM_MOD_N_MATCH=24
# KV on GPU, quantized. Keep K == V so Flash Attention stays enabled.
CACHE_TYPE_K=q8_0
CACHE_TYPE_V=q8_0
# Batch sizing: production-verified values (2026-08-14).
BATCH_SIZE=8192
UBATCH_SIZE=256
# Sampling — Qwen3.8 thinking-mode defaults.
TEMPERATURE=1.0
TOP_K=20
TOP_P=0.95
MIN_P=0.0
THREADS=12
THREADS_HTTP=1
# 10 (debug): prints per-layer device assignment so the split is verifiable
# in the log. Drop to 4 for quiet production use.
VERBOSITY=10
# 1: --direct-io model load. Required in practice for large models loaded
# over RPC (load hangs otherwise; visorcraft Strix RPC finding).
DIRECT_IO=1
scripts/frank-pp-head.service
+16
View File
@@ -0,0 +1,16 @@
[Unit]
Description=llama.cpp Qwen3.8-27B pipeline-parallel head (Strix + RPC to Dankserver)
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=dank
Group=dank
Environment="FRANK_PP_HEAD_ENV_FILE=/home/dank/vllmScripts/scripts/frank-pp-head.env"
ExecStart=/home/dank/vllmScripts/scripts/serve-pp-head.sh
Restart=on-failure
RestartSec=10
[Install]
WantedBy=multi-user.target
scripts/frank-pp-worker.env
+30
View File
@@ -0,0 +1,30 @@
# Worker tunables — Qwen3.8-27B pipeline-parallel RPC worker (Dankserver).
#
# The worker is the REMOTE side of the split. It exposes the RTX 3090 to the
# head on Dankclaw as an RPC device. It runs from the b10433 CUDA+RPC build
# in /home/dank/llama-cpp-rpc (worktree of /home/dank/llama.cpp).
#
# WARNING: do not name any variable here TEMP, HOST, or GPU_MAX_HEAP_SIZE.
# The ROCm HIP runtime reads those bare env vars (see the 2026-08-13 TEMP
# incident in docs/deployment.md).
# Bind address and port for ggml-rpc-server. Must be the LAN address the head
# can reach; the default bind (127.0.0.1) refuses remote clients.
RPC_BIND=192.168.0.235
RPC_PORT=50052
# Device to expose. CUDA0 = the RTX 3090.
DEVICE=CUDA0
# Threads for the worker's CPU device (server has 8 cores; leave 1 for the OS
# and the running frank-default-small service).
THREADS=7
# 1: enable the local tensor file cache (~/.cache/llama.cpp/rpc/). The head
# streams the 31.5 GB model to the worker on first load; with the cache, later
# loads skip the transfer.
CACHE=1
# Worker binary. Must be the SAME llama.cpp build as the head (b10433 /
# 9b05354ec). Version skew hangs the RPC handshake.
WORKER_BIN=/home/dank/llama-cpp-rpc/build/bin/ggml-rpc-server
scripts/frank-pp-worker.service
+15
View File
@@ -0,0 +1,15 @@
[Unit]
Description=llama.cpp Qwen3.8-27B pipeline-parallel RPC worker (RTX 3090)
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=dank
Group=dank
ExecStart=/home/dank/vLLM/scripts/serve-pp-worker.sh
Restart=on-failure
RestartSec=10
[Install]
WantedBy=multi-user.target
scripts/pp-bench.py
Executable
+103
View File
@@ -0,0 +1,103 @@
#!/usr/bin/env python3
"""pp-bench.py — drive the PP head API and report server-side timings.
Runs on the HEAD machine (Dankclaw). Targets the running PP head
(serve-pp-head.sh, port from argument). Builds a ~Nk-token prose prompt,
sends one warmup completion, then REPS measured completions, and reads the
authoritative `slot print_timing` lines from the head log.
Usage:
python3 pp-bench.py <prompt_k> <out_tokens> <reps> <port> <head_log>
"""
import json
import re
import sys
import time
import urllib.request
prompt_k, out_tokens, reps, port, head_log = sys.argv[1:6]
out_tokens = int(out_tokens)
reps = int(reps)
target = int(prompt_k) * 1000
# ~38 tokens per 160-char prose unit; size the filler to hit the target.
unit = ("The careful study of natural language reveals patterns of meaning "
"that emerge from the deliberate arrangement of words and phrases. ")
approx = max(1, target // 38)
prompt = unit * approx + (
"\n\nNow, in a detailed and flowing paragraph, "
"continue this essay about the philosophy of "
"language and its influence on modern computation. "
)
def req(n):
body = json.dumps({"prompt": prompt, "max_tokens": n,
"temperature": 0.0, "stream": False}).encode()
r = urllib.request.Request(
f"http://127.0.0.1:{port}/v1/completions",
data=body, headers={"Content-Type": "application/json"})
with urllib.request.urlopen(r, timeout=7200) as resp:
return json.load(resp)
def log_timings():
try:
lines = open(head_log, encoding="utf-8", errors="ignore").read().splitlines()
except OSError:
return []
out = []
for l in lines:
if re.search(r"prompt eval time|eval time|draft acceptance", l):
out.append(l.strip())
return out
GEN_RE = re.compile(
r"eval time =\s+([\d.]+) ms /\s+(\d+) tokens.*?([\d.]+) tokens per second")
PRE_RE = re.compile(
r"prompt eval time =\s+([\d.]+) ms /\s+(\d+) tokens.*?([\d.]+) tokens per second")
ACC_RE = re.compile(r"draft acceptance = ([\d.]+)")
# warmup (first run pays graph-compile / JIT costs)
warm = req(16)
print(f"warmup: prompt_tokens={warm['usage']['prompt_tokens']} "
f"completion_tokens={warm['usage']['completion_tokens']}")
gen_rates, prefill_rates, accepts = [], [], []
for i in range(reps):
t0 = time.time()
out = req(out_tokens)
dt = time.time() - t0
u = out["usage"]
pt, ct = u["prompt_tokens"], u["completion_tokens"]
gen = pre = acc = None
for l in log_timings():
if "prompt eval" not in l:
m = GEN_RE.search(l)
if m:
gen = float(m.group(3))
m = PRE_RE.search(l)
if m:
pre = float(m.group(3))
m = ACC_RE.search(l)
if m:
acc = float(m.group(1))
gen_rates.append(gen if gen is not None else ct / dt)
if pre is not None:
prefill_rates.append(pre)
if acc is not None:
accepts.append(acc)
print(f"rep {i + 1}: prompt={pt} gen={ct} wall={dt:.1f}s "
f"gen={gen_rates[-1]:.2f} t/s "
f"prefill={f'{pre:.1f}' if pre is not None else 'n/a'} t/s "
f"acceptance={f'{acc:.3f}' if acc is not None else 'n/a'}")
print("== RESULT pp ==")
print(f"avg gen tok/s = {sum(gen_rates) / len(gen_rates):.2f}")
if prefill_rates:
print(f"avg prefill t/s = {sum(prefill_rates) / len(prefill_rates):.1f}")
if accepts:
print(f"avg acceptance = {sum(accepts) / len(accepts):.3f}")
for l in log_timings()[-3:]:
print(" " + l)
scripts/pp-test.sh
Executable
+195
View File
@@ -0,0 +1,195 @@
#!/usr/bin/env bash
# pp-test.sh — Qwen3.8-27B pipeline-parallel (llama.cpp RPC) test orchestrator.
#
# Runs Qwen3.8-27B UD-Q8_K_XL split across Dankclaw (head, Strix iGPU) and
# Dankserver (worker, RTX 3090) over the 2.5 GbE LAN. Both boxes run the same
# llama.cpp build (b10433 / 9b05354ec); the worker build lives in
# /home/dank/llama-cpp-rpc on Dankserver.
#
# BENCHMARKING POLICY (AGENTS.md): this test STOPS both production services
# and restores them when done. Do not run it while users are mid-stream.
#
# Usage (from any box with ssh access to both targets, repo checked out):
# ./scripts/pp-test.sh preflight
# ./scripts/pp-test.sh start # stop prod, start PP head+worker
# ./scripts/pp-test.sh verify # show the layer split + VRAM
# ./scripts/pp-test.sh bench [prompt_k] [out_tokens] [reps]
# ./scripts/pp-test.sh stop-pp # stop PP head+worker only
# ./scripts/pp-test.sh restore # start production services
# ./scripts/pp-test.sh all # start + verify + bench + restore
#
# Sudo: set FRANK_SUDO_PW for headless runs (stop/start of system services).
# FRANK_SUDO_PW=... ./scripts/pp-test.sh start
set -euo pipefail
CLAW="dank@dankclaw"
SERVER="dank@dankserver"
CLAW_REPO="/home/dank/vllmScripts"
SERVER_REPO="/home/dank/vLLM"
HEAD_PORT=7770
HEAD_LOG="/tmp/pp-head.log"
WORKER_LOG="/tmp/pp-worker.log"
RPC_PORT=50052
SUDO="sudo"
if [ -n "${FRANK_SUDO_PW:-}" ]; then
SUDO="printf '%s\n' '$FRANK_SUDO_PW' | sudo -S"
fi
log() { echo "[pp-test] $*"; }
die() { echo "[pp-test] FAIL: $*" >&2; exit 1; }
ssh_claw() { ssh -o BatchMode=yes "$CLAW" "$@"; }
ssh_server() { ssh -o BatchMode=yes "$SERVER" "$@"; }
# ---------------------------------------------------------------------------
# preflight
# ---------------------------------------------------------------------------
do_preflight() {
log "preflight: build identity"
local h_v w_v
h_v=$(ssh_claw '/home/dank/llama.cpp/build/bin/llama-server --version 2>&1 | head -1')
w_v=$(ssh_server '/home/dank/llama-cpp-rpc/build/bin/llama-server --version 2>&1 | head -1')
log " head: $h_v"
log " worker: $w_v"
[[ "$h_v" == *9b05354ec* && "$w_v" == *9b05354ec* ]] || die "build skew (need identical commit 9b05354ec on both sides)"
log "preflight: binaries"
ssh_claw "[ -x $CLAW_REPO/scripts/serve-pp-head.sh ]" || die "missing $CLAW_REPO/scripts/serve-pp-head.sh on claw (git pull?)"
ssh_server "[ -x $SERVER_REPO/scripts/serve-pp-worker.sh ]" || die "missing $SERVER_REPO/scripts/serve-pp-worker.sh on server (git pull?)"
ssh_server "[ -x /home/dank/llama-cpp-rpc/build/bin/ggml-rpc-server ]" || die "missing worker RPC binary on server"
ssh_claw "[ -f /home/dank/models/Qwen3.8-27B-UD-Q8_K_XL.gguf ]" || die "missing model on claw"
log "preflight: network (claw -> server:50052 listener check needs the worker up; testing route + link)"
ssh_claw "ip -4 addr show eno1 | grep -q 192.168.0.143" || die "unexpected claw address"
ssh_server "ip -4 addr show eno1 | grep -q 192.168.0.235" || die "unexpected server address"
ssh_claw "ping -c 2 -W 2 192.168.0.235 >/dev/null" || die "claw cannot reach server"
local rtt
rtt=$(ssh_claw "ping -c 5 -W 2 192.168.0.235 | tail -1" | grep -oE '[0-9.]+/[0-9.]+ ms' | head -1 || true)
log " rtt: ${rtt:-unknown}"
log "preflight: production services"
ssh_claw "$SUDO systemctl is-active frank-default-large" 2>/dev/null || true
ssh_server "$SUDO systemctl is-active frank-default-small" 2>/dev/null || true
log "preflight OK"
}
# ---------------------------------------------------------------------------
# start
# ---------------------------------------------------------------------------
do_start() {
log "stopping production services (both boxes)"
ssh_claw "$SUDO systemctl stop frank-default-large" || die "could not stop frank-default-large"
ssh_server "$SUDO systemctl stop frank-default-small" || die "could not stop frank-default-small"
sleep 5
log "dropping page caches on claw (reliable model load)"
ssh_claw "$SUDO sh -c 'sync; echo 3 > /proc/sys/vm/drop_caches'" || log "WARN: cache drop failed"
log "starting RPC worker on server"
ssh_server "cd $SERVER_REPO && nohup bash scripts/serve-pp-worker.sh > $WORKER_LOG 2>&1 & echo worker_pid=\$!"
for i in $(seq 1 30); do
if ssh_server "ss -tln | grep -q :$RPC_PORT"; then
log "worker listening on :$RPC_PORT"
break
fi
[ "$i" = "30" ] && { ssh_server "tail -20 $WORKER_LOG"; die "worker did not bind :$RPC_PORT"; }
sleep 1
done
log "starting PP head on claw (model load + weight stream: allow ~10 min)"
local head_pid
head_pid=$(ssh_claw "cd $CLAW_REPO && nohup bash scripts/serve-pp-head.sh > $HEAD_LOG 2>&1 & echo \$!")
log "head pid on claw: $head_pid"
for i in $(seq 1 360); do
local st
st=$(ssh_claw "curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:$HEAD_PORT/health 2>/dev/null || true")
if [ "$st" = "200" ]; then
log "head healthy after ~$((i * 5))s"
return 0
fi
if ! ssh_claw "kill -0 $head_pid 2>/dev/null"; then
ssh_claw "tail -30 $HEAD_LOG"
die "head process died during load"
fi
[ $((i % 12)) = 0 ] && log " still loading... ($((i * 5))s)"
sleep 5
done
ssh_claw "tail -30 $HEAD_LOG"
die "head did not become healthy in 30 min"
}
# ---------------------------------------------------------------------------
# verify
# ---------------------------------------------------------------------------
do_verify() {
log "health + device split (from head log)"
ssh_claw "curl -s http://127.0.0.1:$HEAD_PORT/health; echo"
ssh_claw "grep -aE 'using device|assigned to device|MiB free' $HEAD_LOG | head -20 || true"
log "worker VRAM on server:"
ssh_server "nvidia-smi --query-gpu=memory.used --format=csv,noheader"
log "worker log tail:"
ssh_server "tail -5 $WORKER_LOG"
}
# ---------------------------------------------------------------------------
# bench
# ---------------------------------------------------------------------------
do_bench() {
local PROMPT_K="${1:-4}"
local OUT_TOKENS="${2:-256}"
local REPS="${3:-2}"
log "bench: prompt ~${PROMPT_K}k tokens, out=${OUT_TOKENS}, reps=${REPS} (on claw, via :$HEAD_PORT)"
ssh_claw "python3 $CLAW_REPO/scripts/pp-bench.py $PROMPT_K $OUT_TOKENS $REPS $HEAD_PORT $HEAD_LOG"
}
# ---------------------------------------------------------------------------
# stop / restore
# ---------------------------------------------------------------------------
do_stop_pp() {
log "stopping PP head (claw) and worker (server)"
ssh_claw "pkill -f 'serve-pp-head.sh' 2>/dev/null; pkill -f 'llama-server.*frank-pp-large' 2>/dev/null; true"
ssh_server "pkill -x ggml-rpc-server 2>/dev/null; true"
sleep 3
ssh_server "nvidia-smi --query-gpu=memory.used --format=csv,noheader"
}
do_restore() {
log "starting production services"
ssh_server "$SUDO systemctl start frank-default-small" || die "could not start frank-default-small"
ssh_claw "$SUDO systemctl start frank-default-large" || die "could not start frank-default-large"
log "waiting for health (model load takes minutes; up to ~17 min cold on claw)"
for i in $(seq 1 300); do
local a b
a=$(ssh_claw "curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:7770/health 2>/dev/null || true")
b=$(ssh_server "curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:7770/health 2>/dev/null || true")
if [ "$a" = "200" ] && [ "$b" = "200" ]; then
log "both production services healthy after ~$((i * 4))s"
return 0
fi
[ $((i % 15)) = 0 ] && log " claw=$a server=$b ... ($((i * 4))s)"
sleep 4
done
die "production services did not both become healthy"
}
# ---------------------------------------------------------------------------
case "${1:-}" in
preflight) do_preflight ;;
start) do_start ;;
verify) do_verify ;;
bench) shift; do_bench "$@" ;;
stop-pp) do_stop_pp ;;
restore) do_restore ;;
all)
do_preflight
do_start
do_verify
do_bench
do_restore
;;
*)
echo "Usage: $0 {preflight|start|verify|bench [prompt_k] [out] [reps]|stop-pp|restore|all}"
exit 1
;;
esac
scripts/serve-pp-head.sh
Executable
+140
View File
@@ -0,0 +1,140 @@
#!/usr/bin/env bash
# Serve the Qwen3.8-27B pipeline-parallel head on Dankclaw.
#
# Runs llama-server (b10433) with --rpc pointing at the worker on
# Dankserver. The model is split pipeline-style across [RPC0, iGPU]:
# layers run in order, hidden states cross the ethernet boundary once per
# token step.
#
# IMPORTANT: no --device flag. The RPC device only joins the model's device
# list through default device selection; an explicit --device silently drops
# it. See the header of frank-pp-head.env.
#
# Tunables come from frank-pp-head.env in this directory unless the
# FRANK_PP_HEAD_ENV_FILE environment variable overrides that path.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ENV_FILE="${FRANK_PP_HEAD_ENV_FILE:-$SCRIPT_DIR/frank-pp-head.env}"
if [[ -f "$ENV_FILE" ]]; then
set -a
# shellcheck disable=SC1090
source "$ENV_FILE"
set +a
fi
# --- defaults (used when the env file does not set them) -------------------
: "${MODEL_DIR:=/home/dank/models}"
: "${MODEL_FILE:=Qwen3.8-27B-UD-Q8_K_XL.gguf}"
: "${HEAD_BIN:=/home/dank/llama.cpp/build/bin/llama-server}"
: "${ALIAS:=frank-pp-large}"
: "${BIND_ADDR:=0.0.0.0}"
: "${LISTEN_PORT:=7770}"
: "${RPC_SERVERS:=192.168.0.235:50052}"
: "${TENSOR_SPLIT:=1,1}"
: "${CTX_SIZE:=131072}"
: "${PARALLEL:=1}"
: "${SPEC_NONE:=0}"
: "${SPEC_DEFAULT:=1}"
: "${MTP:=0}"
: "${SPEC_DRAFT_N_MAX:=4}"
: "${SPEC_DRAFT_MODEL:=}"
: "${SPEC_NGRAM_MOD_N_MIN:=48}"
: "${SPEC_NGRAM_MOD_N_MAX:=64}"
: "${SPEC_NGRAM_MOD_N_MATCH:=24}"
: "${CACHE_TYPE_K:=q8_0}"
: "${CACHE_TYPE_V:=q8_0}"
: "${BATCH_SIZE:=8192}"
: "${UBATCH_SIZE:=256}"
: "${TEMPERATURE:=1.0}"
: "${TOP_K:=20}"
: "${TOP_P:=0.95}"
: "${MIN_P:=0.0}"
: "${THREADS:=12}"
: "${THREADS_HTTP:=1}"
: "${VERBOSITY:=4}"
: "${DIRECT_IO:=1}"
MODEL="$MODEL_DIR/$MODEL_FILE"
if [[ ! -f "$MODEL" ]]; then
echo "missing required file: $MODEL" >&2
exit 1
fi
if [[ ! -x "$HEAD_BIN" ]]; then
echo "missing head binary: $HEAD_BIN" >&2
exit 1
fi
# --- RPC endpoints -----------------------------------------------------------
RPC_ARGS=()
for s in ${RPC_SERVERS//,/ }; do
RPC_ARGS+=(--rpc "$s")
done
# --- split fractions ---------------------------------------------------------
SPLIT_ARGS=()
if [[ -n "${TENSOR_SPLIT}" ]]; then
SPLIT_ARGS+=(--tensor-split "${TENSOR_SPLIT}")
fi
# --- speculative decoding -----------------------------------------------------
SPEC_ARGS=()
if [[ "${SPEC_NONE}" == "1" ]]; then
SPEC_ARGS+=(--spec-type none)
else
if [[ "${SPEC_DEFAULT}" == "1" ]]; then
if [[ "${MTP}" == "1" ]]; then
# MTP first (bundled in-model head by default, or a standalone
# draft file when SPEC_DRAFT_MODEL is set), then the ngram-mod
# fallback. Order matters: --spec-type appends to the impl chain.
SPEC_ARGS+=(--spec-type draft-mtp)
SPEC_ARGS+=(--spec-draft-n-max "${SPEC_DRAFT_N_MAX}")
if [[ -n "${SPEC_DRAFT_MODEL}" ]]; then
if [[ ! -f "${SPEC_DRAFT_MODEL}" ]]; then
echo "SPEC_DRAFT_MODEL file missing: ${SPEC_DRAFT_MODEL}" >&2
exit 1
fi
SPEC_ARGS+=(--spec-draft-model "${SPEC_DRAFT_MODEL}")
fi
fi
SPEC_ARGS+=(--spec-default)
SPEC_ARGS+=(--spec-ngram-mod-n-min "${SPEC_NGRAM_MOD_N_MIN}")
SPEC_ARGS+=(--spec-ngram-mod-n-max "${SPEC_NGRAM_MOD_N_MAX}")
SPEC_ARGS+=(--spec-ngram-mod-n-match "${SPEC_NGRAM_MOD_N_MATCH}")
else
echo "SPEC_NONE=0 requires SPEC_DEFAULT=1" >&2
exit 1
fi
fi
# --- load mode ----------------------------------------------------------------
LOAD_ARGS=()
if [[ "${DIRECT_IO}" == "1" ]]; then
LOAD_ARGS+=(--direct-io)
fi
exec "$HEAD_BIN" \
--model "$MODEL" \
"${RPC_ARGS[@]}" \
"${SPLIT_ARGS[@]}" \
"${SPEC_ARGS[@]}" \
"${LOAD_ARGS[@]}" \
--gpu-layers all \
--fit off \
--cache-type-k "$CACHE_TYPE_K" \
--cache-type-v "$CACHE_TYPE_V" \
--ctx-size "$CTX_SIZE" \
--parallel "$PARALLEL" \
--batch-size "$BATCH_SIZE" \
--ubatch-size "$UBATCH_SIZE" \
--temp "$TEMPERATURE" \
--top-k "$TOP_K" \
--top-p "$TOP_P" \
--min-p "$MIN_P" \
--threads "$THREADS" \
--threads-http "$THREADS_HTTP" \
--alias "$ALIAS" \
-lv "$VERBOSITY" \
--host "$BIND_ADDR" \
--port "$LISTEN_PORT"
scripts/serve-pp-worker.sh
Executable
+47
View File
@@ -0,0 +1,47 @@
#!/usr/bin/env bash
# Serve the Qwen3.8-27B pipeline-parallel RPC worker on Dankserver.
#
# Runs ggml-rpc-server from the b10433 CUDA+RPC build and exposes the RTX
# 3090 as a remote device for the head on Dankclaw.
#
# Tunables come from frank-pp-worker.env in this directory unless the
# FRANK_PP_WORKER_ENV_FILE environment variable overrides that path.
#
# Security note (upstream): the RPC server is "fragile and insecure". No
# authentication, no encryption. Keep it on the trusted LAN only.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
ENV_FILE="${FRANK_PP_WORKER_ENV_FILE:-$SCRIPT_DIR/frank-pp-worker.env}"
if [[ -f "$ENV_FILE" ]]; then
set -a
# shellcheck disable=SC1090
source "$ENV_FILE"
set +a
fi
# --- defaults (used when the env file does not set them) -------------------
: "${RPC_BIND:=192.168.0.235}"
: "${RPC_PORT:=50052}"
: "${DEVICE:=CUDA0}"
: "${THREADS:=7}"
: "${CACHE:=1}"
: "${WORKER_BIN:=/home/dank/llama-cpp-rpc/build/bin/ggml-rpc-server}"
if [[ ! -x "$WORKER_BIN" ]]; then
echo "missing worker binary: $WORKER_BIN" >&2
echo "Build it: cd /home/dank/llama-cpp-rpc && cmake -B build -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DGGML_CUDA_FA_ALL_QUANTS=ON -DGGML_RPC=ON -DCMAKE_BUILD_TYPE=Release && cmake --build build --config Release -- -j 8" >&2
exit 1
fi
ARGS=()
if [[ "${CACHE}" == "1" ]]; then
ARGS+=(--cache)
fi
exec "$WORKER_BIN" \
--host "$RPC_BIND" \
--port "$RPC_PORT" \
--device "$DEVICE" \
--threads "$THREADS" \
"${ARGS[@]}"